Methods and apparatus for three-dimensional (3D) imaging

ABSTRACT

A method of imaging a scene includes estimating multiple three-dimensional (3D) representations, each of which corresponds to a respective portion of the scene. Neighboring portions of the scene area are at least partially overlapping. Each 3D representation is estimated by illuminating the respective portion of the scene with a light burst including multiple light pulses, after which multiple point clouds are generated by detecting photons reflected or scattered from the respective portion of the scene using a focal plane array. Data points in the point clouds represent a distance between the focal plane array and a scene point in the respective portion of the scene. The 3D representation is then estimated based on the multiple point clouds via coincidence processing. The method then generates a 3D image of the scene based on the multiple 3D representations.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application is continuation application of U.S. application Ser.No. 15/147,854, now U.S. Pat. No. 9,915,733, filed May 5, 2016, which isa bypass continuation of International Application No.PCT/US2015/046682, filed Aug. 25, 2015, which in turn claims priority,under 35 U.S.C. § 119(e), from: U.S. Application No. 62/042,303, filedAug. 27, 2014, and entitled “PHOTON-TIMING DETECTOR ARRAY WITH EMBEDDEDPROCESSOR”; U.S. Application No. 62/042,321, filed Aug. 27, 2014, andentitled “OPTICAL FIELD-OF-VIEW MULTIPLEXER”; U.S. Application No.62/041,713, filed Aug. 26, 2014, and entitled “PHOTON-TIMING DETECTORARRAY WITH CONFIGURABLE REGION OF INTEREST”; and U.S. Application No.62/041,755, filed Aug. 26, 2014, and entitled “MICROLADAR SYSTEMARCHITECTURE.” Each of these applications is hereby incorporated hereinby reference in its entirely.

GOVERNMENT SUPPORT

This invention was made with government support under Contract No.FA8721-05-C-0002 awarded by the U.S. Air Force. The government hascertain rights in the invention.

BACKGROUND

There are generally two types of remote sensing technologies: passivesensing and active sensing. In passive sensing, images or otherrepresentations of a target are created by detecting radiation that isgenerated by an external source, such as the sun. In contrast, activesensing technologies not only detect radiation reflected or scattered bya target but also generate the radiation to illuminate the target forsubsequent detection.

Light Detection and Ranging (LIDAR, also known as LADAR) is an activesensing technique that involves emitting light (e.g., pulses from alaser) and detecting the reflected or scattered light. LIDAR typicallymeasures the time-of-flight (i.e., the time it takes for the pulse totravel from the transmitter to the target, be reflected, and travel backto the sensor), which can be used to derive ranges (or distances) to thetarget which reflects or scatters the light. In this manner, LIDAR isanalogous to radar (radio detecting and ranging), except that LIDAR isbased on optical waves instead of radio waves.

LIDAR can be airborne or ground-based. Airborne LIDAR typically collectsdata from airplanes looking down and covering large areas of the ground.LIDAR can also be conducted from ground-based stationary and mobileplatforms. Ground-based LIDAR techniques can be beneficial in producinghigh accuracies and point densities, thus permitting the development ofprecise, realistic, three-dimensional representations of scenes such asrailroads, roadways, bridges, buildings, breakwaters, or shorelinestructures.

SUMMARY

Embodiments of the present invention include methods and apparatus forthree-dimensional imaging of a scene. In one example, a method ofimaging a scene includes estimating a first three-dimensional (3D)representation of a first portion of the scene. The estimating of thefirst 3D representation includes illuminating the first portion of thescene with a first light burst comprising a first plurality of lightpulses, the first light burst having a first burst durationsubstantially equal to or less than 1 millisecond. The estimation of thefirst 3D representation also includes generating a first plurality ofpoint clouds, corresponding to the first plurality of the light pulses,by detecting photons reflected or scattered from the first portion ofthe scene using a focal plane array. A first data point in the firstplurality of point clouds represents a first distance between the focalplane array and a first scene point in the first portion of the scene.The estimation of the first 3D representation also includes estimatingthe first 3D representation of the first portion of the scene based atleast in part on the first plurality of point clouds. The method alsoincludes estimating a second 3D representation of a second portion ofthe scene at least partially overlapping with the first portion of thescene. The estimating of the second 3D representation includesilluminating the second portion of the scene with a second light burstcomprising a second plurality of light pulses, the second light bursthaving a second burst duration substantially equal to or less than 1millisecond. The estimating of the second 3D representation alsoincludes generating a second plurality of point clouds, corresponding tothe second plurality of the light pulses, by detecting photons reflectedor scattered from the second portion of the scene using the focal planearray. A second data point in the second plurality of point cloudsrepresents a second distance between the focal plane array and a secondscene point in the second portion of the scene. The estimation of thesecond 3D representation further includes estimating the second 3Drepresentation of the second portion of the scene based at least in parton the second plurality of point clouds. The method then generates athree-dimensional image of the scene based at least in part on the first3D representation of the first portion of the scene and the second 3Drepresentation of the second portion of the scene.

In another example, an apparatus for generating a 3D image of a sceneincludes a light source to illuminate the scene with light pulses at arepetition rate greater than 10 kHz. The apparatus also includes adetector array to detect photons reflected or scattered by the scene soas to generate a detected data set representing respective arrival timesand arrival angles of the photons reflected or scattered by the scene.An embedded processor is operably coupled to the detector array togenerate a processed data set by removing at least one redundant datapoint and/or at least one noise data point from the detected data set. Aprocessor is communicatively coupled to the embedded processor via adata link to receive the processed data set and generate the 3D image ofthe scene based on the processed data set.

In another example, a method of generating a representation of a sceneincludes illuminating a first portion of the scene with at least onefirst light pulse, followed by acquiring at least one first raw pointcloud from photons reflected or scattered from the first portion of thescene using a focal plane array. A first data point in the at least onefirst raw point cloud represents a first distance between the focalplane array and a first scene point in the first portion of the scene.The method also includes removing, with a first processor, at least oneof a first noise data point or a first redundant data point from atleast one first raw point cloud so as to generate a first processedpoint cloud having fewer data points that the at least one first rawpoint cloud. The method further includes conveying the first processedpoint cloud to a second processor.

In another example, a method of imaging a scene includes illuminatingthe scene with an illumination light pulse, followed by acquiring anillumination point cloud, using a focal plane array, from photons in theillumination light pulse reflected or scattered from the scene. An areaof interest is determined in the focal plane array based at least inpart on the illumination point cloud, wherein the area of interest issmaller than an area of the focal plane array. The method also includesilluminating the scene with a plurality of signal light pulses. For eachsignal light pulse in the plurality of signal light pulses, a respectivesignal point cloud generated by at least one respective signal photonreflected or scattered by the scene is acquired from the area ofinterest in the focal plane array. The method further includesgenerating a three-dimensional image of at least a portion of the scenebased at least in part on a plurality of signal point clouds from theplurality of signal pulses.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts discussed in greater detail below (provided suchconcepts are not mutually inconsistent) are contemplated as being partof the inventive subject matter disclosed herein. In particular, allcombinations of claimed subject matter appearing at the end of thisdisclosure are contemplated as being part of the inventive subjectmatter disclosed herein. It should also be appreciated that terminologyexplicitly employed herein that also may appear in any disclosureincorporated by reference should be accorded a meaning most consistentwith the particular concepts disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The skilled artisan will understand that the drawings primarily are forillustrative purposes and are not intended to limit the scope of theinventive subject matter described herein. The drawings are notnecessarily to scale; in some instances, various aspects of theinventive subject matter disclosed herein may be shown exaggerated orenlarged in the drawings to facilitate an understanding of differentfeatures. In the drawings, like reference characters generally refer tolike features (e.g., functionally similar and/or structurally similarelements).

FIG. 1A shows a schematic of a LIDAR system including an embeddedprocessor.

FIG. 1B is a block diagram showing an architecture of the LIDAR systemin FIG. 1A.

FIG. 1C shows an example LIDAR system including a field-programmablegate array (FPGA).

FIG. 2A illustrates 3D imaging techniques including on-board dataprocessing.

FIGS. 2B and 2C are examples of a raw point cloud and a processed pointcloud respectively.

FIG. 3 illustrates LIDAR methods including on-board data processing.

FIG. 4 illustrates LIDAR methods including sequential imageregistration.

FIG. 5 illustrates LIDAR methods including sequential image registrationto cover a wide-area scene.

FIG. 6 illustrates LIDAR techniques including configurable field ofinterest.

FIG. 7 shows a schematic of an example optical imaging system suitablefor use in a LIDAR system like the one shown in FIG. 1A.

FIG. 8 shows a schematic of an optical system including an opticalfield-of-view (FOV) multiplexer suitable for use in a LIDAR system likethe one shown in FIG. 1A.

FIG. 9 shows an example of detector arrays that are tiled across a laserillumination pattern that has been imaged onto a focal plane like thefocal plane in the LIDAR system shown in FIG. 1A.

FIG. 10 illustrates a data flow for 3D imaging using a LIDAR system likethe one shown in FIGS. 1A and 1B.

FIGS. 11A-11C shows example results for coarse registration of images ofrespective 100 m² areas that overlap with neighboring areas by about 70%and that were acquired by an airborne LIDAR system like the one shown inFIG. 1A.

DETAILED DESCRIPTION

Overview

A LIDAR system, using an airborne LIDAR system as an example, caninclude three major modules: a laser scanner, a georeferencing module,and a computer processing module.

A laser scanner can be further divided into three sub-components: anopto-mechanical scanner, a ranging unit, and a control processing unit.The opto-mechanical scanner can be a precision-engineered device thatcan provide a consistent stream of laser pulses, which are employed toilluminate the scene to be imaged. These laser pulses are typically sentinto the ranging unit, where the laser pulses are reflected off a mirror(either rotating or scanning) and thereby transmitted to the target. Theranging unit also includes an electro-optical receiver to record thelaser pulses that are returned from the scene, including recording thetravel time of the laser pulses and communicating this data to thecontrol processing unit. The repetition rate of this process can be ashigh as 100 kHz to 200 kHz.

A laser scanner by itself typically is not capable of recognizing orcalculating coordinates of the reflected laser points. A georeferencingmodule is normally employed in conventional LIDAR systems to extrapolatea 3D coordinate from the range signal produced by the laser scanner soas to locate the scanner with respect to a coordinate system andextrapolate the range (distance) measured to a precise location on theground. Georeferencing can be a direct measurement of the receiverposition (X, Y, Z) and orientation (roll, pitch, and heading) withrespect to the ground.

One example of georeferencing can be implemented by a Global PositioningSystem (GPS) and an inertial navigation system. GPS is a satellite-basedtechnology embedded in many devices, from cell phones to automobiles.Similarly, LIDAR systems can use GPS technology to derive the precise X,Y, Z position of the receiver in three-dimensional space by, forexample, mounting a GPS receiver on a moving platform, such as a vehicleor airplane. The GPS receiver used in LIDAR systems can be robustdual-frequency receivers with differential post-processing.

The inertial navigation in the georeferencing module can be based on twophysics principles: 1) an object spinning rapidly tends to keep itsrelative orientation in space; and 2) on Earth, a rapidly spinningobject typically aligns itself with gravity. In example implementations,the inertial navigation can be realized by an Inertial Measurement Unit(IMU) that includes three inertial gyroscopes that are aligned to thelaser head in a LIDAR system, in which case the angular rotations of thereceiver from vertical can be measured.

The IMU can also include accelerometers to measure the velocity (e.g.,derived from acceleration measurement). An accelerometer, forillustrative purposes only, may be visualized as a weight suspended on astring from the top center of a container. Under quick movement of thecontainer, the container normally moves first and the weight catches upwith the movement with a delay (e.g., a fraction of a second). With ahigh precision clock and the ability to measure angles of the movement,an accelerometer can measure both the speed and direction of movement ofthe LIDAR receiver. However, the accuracy of the measurements fromaccelerometers tends to decrease as a function of time because ofmechanical limitations and the external influence of gravity. Periodicupdating of position can be provided by the GPS every 0.5 or 1 second.

The GPS/IMU system for a LIDAR system can be initialized on the groundwith respect to a base station. During the flight mission, the LIDARsystem normally stays within 50 miles of the reference base station inorder to maintain the accuracy. Upon completion of the flight, the LIDARsystem and the reference base station are reinitialized to provideclosure.

The computer processing module in a LIDAR system coordinates theactivity of each individual component (e.g., laser scanner, GPS, andIMU). For example, the computer processing module can integrate the datafrom each component into usable and accurate range information (e.g.,elevation on the ground). The data from each component includes, forexample, the time information of the reflected laser pulses generated bythe laser scanner is to generate laser pulses and time the reflections,the positioning data from the GPS, and the angular motion and velocitydata from the IMU.

Compared to some conventional remote sensing techniques, such as aerialphotographs, LIDAR techniques can have several advantages. For example,LIDAR can rapidly measure the Earth's surface, e.g., at sampling ratesgreater than 150 KHz (i.e., 150,000 pulses per second). The resultingproduct is a densely spaced network of highly accurate georeferencedelevation points (often referred to as a point cloud), which can be usedto generate three-dimensional representations of the Earth's surface.

LIDAR can also operate with high accuracy. Typically, LIDAR-derivedelevations can have absolute accuracies of about 6 to 12 inches (15 to30 centimeters) for older data and 4 to 8 inches (10 to 20 centimeters)for more recent data; relative accuracies (e.g., heights of roofs,hills, banks, and dunes) can be even better.

LIDAR can also “see under trees” when, for example, acquiring elevationdata from above the Earth's surface (e.g., from airplanes orsatellites). As long as some photons of the laser pulses in LIDAR canreach the ground through the canopy and reflect or scatter back to thereceiver, the reflected photons can be employed to derive the groundinformation. Practically, if one can see the sky through the trees atcertain location, then that location can be measured with LIDAR.

To further explore the benefit of LIDAR concept, it can be desirable toreduce the size, weight, and power (SWaP) of the systems that implementthe LIDAR concept. A conventional LIDAR system typically includes atleast a laser source, a detector, a precision opto-mechanical scanner,and an IMU. With the development of semiconductor technologies, lasersand detectors have advanced in miniaturization. However, theopto-mechanical scanners and IMU remain bulky in most existing LIDARsystems. Therefore, the total SWaP of a LIDAR system is typicallydominated by the precision opto-mechanical scanners and the IMU that areemployed to create sharp imagery.

LIDAR Systems Including Embedded Processors

LIDAR techniques (and the systems implementing the techniques) caninclude one or more of the following features to further reduce thetotal SWaP:

-   -   An array of detectors can be employed to derive relative        alignment of pixels across a sizable image chip;    -   The system can be operated so that a sizeable image chip can be        acquired faster than any blurring of the image can occur;    -   The system can be operated so that image chips are collected        fast enough so the array field-of-view moves between chips by        less than the FOV width, thereby ensuring significant        overlapping between subsequent chips;    -   The detectors can be operated in a photon-counting mode so that        a minimal number of detected photons can be sufficient in making        a range measurement (e.g., 7-15 photons), and therefore low        power lasers and/or small receiver apertures can be used;    -   Processing of the raw point clouds into estimates of scene        content can be simplified to the point of implementation on        low-power FPGAs since the processing can be done in        angle-angle-range space and is therefore local and pipelineable;    -   Individual image chips can be pieced together into a mosaic of a        useful size using 3D registration, therefore eliminating the        need for precision opto-mechanics and platform position sensors;    -   Compact pulsed sources, such as a CW-pumped passively Q-switched        microchip laser or fiber laser, can be employed to provide        sufficient power since the power-aperture requirements can be        small;    -   Temperature control of the laser can be provided by passive        means, and may be simplified by the addition of passive        wavelength stabilization elements in the laser pump diodes;    -   The detector array can function without the need of temperature        control. Instead, the operating voltage of the detectors can be        adjusted to optimize performance at the measured device        temperature.

The resulting compact LIDAR systems can find broad applications in avariety of fields including, but are not limited to vision system forautonomous navigation of robots; surveillance sensors for small UAVs;navigation in GPS-denied environments; autonomous navigation andcollision avoidance sensors in automobiles and aircraft; driverassistance systems in automobiles; line-of-sight analysis; helicopterlanding zone determination; trafficability maps; route finding; disasterassessment (flood zone determination); pipe- and power-lineinfrastructure monitoring; construction; forestry; mining; cued 3Dsensing for object identification; and multi-mode receivers for opticalcommunications.

FIG. 1A shows a LIDAR system 100 that can address, at least partially,the challenges to further reduce the total SWaP of LIDAR systems. Thisapproach can use a pulsed laser to illuminate the scene and achieve thedesired range resolution. In one example, the laser can be one or a fewpassively Q-switched microchip lasers. To simplify thermal control, thewavelength of the pump diodes may be grating-stabilized, therebywidening the allowable temperature window. To simplify electricalcontrol the diodes may be operated in the CW pumping mode during theburst, without the necessity of fast, high-current electronics. Theaverage PRF can be tweaked by adjusting the diode pump current. Themicrochip laser can be mounted in very close proximity to the pump diodeemission facet, perhaps separated by mode-matching optics. In anotherexample, the laser can be a pulsed fiber laser, which can have higherelectrical-to-optical conversion efficiency, and therefore longerachievable ranges for a given platform power budget. Fiber lasers can bemade very compact, and even fabricated as waveguides on an integratedoptical device. In yet another example, the laser can bedirectly-modulated semiconductor lasers, which can have even higherefficiency (e.g., 10¹-10² times higher PRFs, compared to microchips).

The LIDAR system 100 includes a light source 110 configured to transmitlight pulses 10 that can illuminate a scene 20. At least a portion ofthe scene 20 reflects or scatters the incident light pulses 10 back tothe LIDAR system 100. An imaging optic 170 collects at least a portionof the return pulses 15 and transmits the return pulses 15 to a focalplane array 130 comprising a plurality of sensing elements 132 (threeare shown in FIG. 1A). Photons in the return pulses 15 can be detectedand converted into electrical signals by the focal plane array 130. Thefocal plane array 130 and the light source 110 are communicativelycoupled by a timing unit 120, which helps the focal plane array 130 torecord the time of flight of the photons reflected or scattered back bythe scene 20. An embedded processor 140 is operably coupled to the focalplane array 130 to promptly process raw data generated by the focalplane array 130. The processed data is then conveyed to a secondprocessor 150 via a data link 160 for further processing (e.g.,registration of 3D images of the scene).

For illustrative purposes only, the LIDAR system 100 shown in FIG. 1 isan airborne LIDAR which can be, for example, mounted on an airplane or ahelicopter so as to image a wide area of scene. In another example, theLIDAR system 100 can be ground-based, in which case the LIDAR system 100can be mounted on a vehicle or a robot to image a scene. In yet anotherexample, the LIDAR system 100 can be mounted on a stationary platformsuch as a tripod or mast. In yet another example, the LIDAR system 100can be a portable and/or a hand-held device, which can be used inenvironments which have constraint in accessibility to vehicle.

In one example, the LIDAR system 100 can be employed in short-rangesystems that operate at ranges of less than 100 meters with panoramicscanning, such as mapping building interiors or ranging small objects.In another example, the LIDAR system 100 can be used for medium rangesystems operating at distances of, for example, 100-250 meters. Thesesystems can achieve centimeter range accuracies (e.g. vertical accuracy)in high definition (angular, e.g. horizontal accuracy), such as forsurveying and/or 3D modeling a bridge and dam monitoring. In yet anotherexample, the LIDAR system 100 can be used in long range systems, whichcan measure at distances of up to one kilometer and are frequently usedin open-pit mining and topographic survey applications.

The Light Source:

The light source 110 in the LIDAR system 100 can be either incoherent orcoherent. Incoherent light sources can include, for example, white-lightsources generated by laser-created filaments. Coherent light source caninclude lasers operating at various wavelengths and modes.

It can be helpful, in general, for the light source 110 to have thefollowing properties: high power and/or brightness, short pulsedurations (e.g., 0.1 ns to 10 ns), the capability to modulate the laserlight with a frequency chirp, high collimation, and narrow bandwidth ofthe optical spectrum. High power and/or brightness of the light source110 can better illuminate the scene to be imaged. Short pulse durationscan improve the temporal resolution of the imaging. Frequency chirp ofthe light pulses can be used to construct LIDAR systems operating incontinuous mode. High collimation of the light pulses may increase thenumber of photons that can be reflected or scattered back to thedetector, thereby increasing the photon collection efficiency of theLIDAR system (i.e., fewer photons are required in each illuminationlight pulse to image a scene). Narrow spectral line can be advantageousin LIDAR techniques at least because narrow optical interference filtersusually with 1 nm bandwidth can be mounted in the receiving path tosuppress disturbing background radiation, caused, e.g., by backscatteredsunlight.

The wavelength of the light source 110 can depend on several practicalconsiderations. In one example, the wavelength of the laser source 110can be dependent on the wavelength sensitivity of the detector (e.g.,the focal plane array 130) used in the LIDAR system. Generally speaking,it can be helpful to use a wavelength at which the correspondingdetector is most sensitive. Greater sensitivity lowers the amount ofenergy for each light pulse, or the number of photons in each lightpulse, to achieve a given signal strength. Currently, sensitivedetectors are available at wavelengths between 500 nm and 2200 nm.Therefore, the light source 110 in the LIDAR system can also operatewithin this wavelength range (e.g., semiconductor laser diodes).

In another example, the wavelength of the light source 110 can bedependent on the consideration of safety to eyes of humans (or othercreatures). If higher pulse energy is desired, it can be desirable tooperate the light source 110 at wavelengths in which the eye is lesssusceptible to damage. Generally, eye safety issues can be relaxed atwavelengths below 400 nm or above about 1400 nm because light at thesewavelengths typically do not penetrate human tissue and therefore onlyinteract at the surface of the cornea or skin. At these wavelengths,higher energy levels can be employed without significant risk ofdamaging the eye.

Using a high energy in each light pulse at eye-safe wavelengths can alsoincur several other benefits. First of all, the higher energy in eachpulse can compensate the relatively lower sensitivity of detectors.Second, the maximum range can also be extended, by using higher energyin each pulse, to more than 1500 meters while maintaining the sameranging performance with respect to signal-to-noise-ratio (SNR) andaccuracy. Third, the background radiation caused by the sunlight, whichcan degrade the SNR, can be lower at eye-safe wavelengths (e.g., 1535nm), at least because spectral components at these wavelengths occupyonly a small portion of the sun light, which is mostly visible.

In yet another example, the wavelength of the light source 110 can bedependent on the reflective or scattering properties of the targetsurface. In general, it can be helpful to choose a wavelength at whichthe target surface can have high reflectivity. For example, trees canhave a reflectivity of about 50% at wavelengths between approximately700 nm and 1400 nm, but their reflectivity decreases to about 10%outside this range. Snow and ice, in contrast, have a good reflectivity(>65%) at wavelengths below 1400 nm, but a low reflectivity (<40%) atabout 1400 nm. The reflectivity of asphalt and soil generally is higherat longer wavelengths within the range of 400 nm and 2000 nm.Accordingly, LIDAR systems can operate in the near-infrared region fortopography applications, although some systems operate in the green bandto penetrate water and detect bottom features.

In yet another example, the wavelength of the light source 110 can bedependent on the atmospheric transmission (or absorption). For longrange distances, the atmosphere may absorb a significant portion oflight at certain wavelengths particularly in the near infrared andinfrared region. Therefore, it can be beneficial to use wavelengths oflight that is less absorptive. For short distances, atmospherictransmission can be less influential in selecting the wavelength of thelight source 110.

In yet another example, the light source 110 in the LIDAR system 100 canoperate at multiple wavelengths so as to, for example, cover a sceneincluding more than one surface type or more than one designatedapplication (topography, hydrography, or glaciography), or to retrieveadditional information about the targets, such as, e.g., crop analysisor tree species detection by using several wavelengths in parallel.

Multiple-wavelength operation of the LIDAR system 100 can be achieved inseveral schemes. In one example, the LIDAR system 100 can scan a targetscene 20 with at least two laser sources 110 operating at differentwavelengths during separate flights. Laser sources 110 may be mounted ondifferent platforms or on the same platform that are exchanged betweenflights. In another example, the LIDAR system 100 can implementmulti-wavelength operation via joint scanning of a target scene 20 usingat least two laser sources 110 operating at different wavelengthsmounted on the same platform. In yet another example, the LIDAR system100 can scan the target the scene 20 with at least two substantiallycoaxial laser beams at different wavelengths such that photons ofdifferent wavelengths can be collected almost simultaneously. In yetanother example, multi-wavelength operation of the LIDAR system 100 canbe achieved by using a tunable laser source 110. The LIDAR system 100can scan one portion of the scene 20 using one wavelength and scananother portion of the scene using another wavelength. Additionally oralternatively, the LIDAR system 100 can tune the wavelength of the lightsource 110 during the scanning of a same portion of the scene 20 so asto capture additional information of the scene.

In view of the above considerations, various specific laser systems canbe employed as the light source 110 in the LIDAR system 100. In oneexample, the light source 110 can include a passively Q-switched Nd:YAG(neodymium-doped yttrium-aluminum-garnet) microchip laser operating ateither the fundamental wavelength at 1064 nm or the second harmonic at532 nm. The Nd:YAG laser can be pumped by one or more diode lasers.

In another example, an actively Q-switched Nd:YAG laser can be used togenerate sub-nanosecond pulses at 30-100 kHz repetition rates withsingle- or few-mode operation. Commercially-available amplifiers canboost the pulse energies to millijoule levels or more. The low timingjitter of active Q-switching allows the LIDAR system designer toprecisely control the PRF during system operation. In this way, one mayaccommodate multiple pulses that are traversing the distance between thesystem and the target, while simultaneously ensuring that the brightoutgoing pulses do not dazzle the sensitive photodetectors while theyare sensing the weak return pulses.

In another example, the light source 110 can include a fiber laser,which typically has good spatial and spectral qualities and can beconfigured to operate in continuous, modulated, or pulsed mode. Outputwavelength of fibers lasers are tunable and can be eye-safe. The core ofthe fiber can be doped with rare-earth elements such as erbium (Er),ytterbium (Yb), neodymium (Nd), dysprosium (Dy), praseodymium (Pr),thulium (Tm), holmium (Ho), and different synthesizers of theseelements. Nd- or Yb-doped silica fiber could provide emission around 1μm. Yb-doped silica fiber can be a promising platform for high powerapplications due to the high optical to optical conversion efficiency.Er-doped silica fiber lasers and amplifiers can operate at around 1.55μm. Emission at 2 μm can be achieved by thulium or holmium-doped silicaor germanate fibers.

In yet another example, the light source 110 can include a semiconductorlaser. The semiconductor laser can produce diffraction-limited emissionby, for example, a ridge waveguide having a width of about severalmicrons so as to preferably lase (amplify via stimulated emission) thefundamental mode. The semiconductor laser can also produce spectrallystabilized emission using a Bragg grating integrated into thesemiconductor chip so as to construct a distributed Bragg reflector(DBR) laser or a distributed feedback (DFB) laser. Q-switch techniquescan also be used in semiconductor lasers to produce short laser pulses(e.g., pulse duration shorter than 1 ns) for LIDAR applications.

Semiconductor optical amplifiers, either monolithically or hybridintegrated with the master oscillator, can be used to increase theoutput energy of the laser. The amplifiers can be constructed in amulti-stage configuration, in which case the amplifiers can also beemployed to control the repetition rate, either as pulse picker toselect individual pulses or as optical gate to generate an optical pulseout of a continuous wave (CW) master oscillator with desired spectralproperties. Using a DFB-RW laser operated in continuous wave as masteroscillator and a multi-section tapered amplifier, optical pulses havinga length of 15 ns at a repetition rate of 25 kHz with peak powers up to16 W can be obtained.

In yet another example, the light source 110 can include a semiconductorlaser based on based on two InGaAsP/InP monolithic Master OscillatorPower Amplifiers (MOPAs) operating at, for example, about 1.57 μm. EachMOPA can include a frequency stabilized Distributed Feedback (DFB)master oscillator, a modulator section, and a tapered amplifier. The useof a bended structure (the oscillator, the modulator, and the amplifierare not arranged along a straight line) may avoid undesired feedback andthus provide good spectral properties together with high output powerand good beam quality.

In yet another example, the light source 110 can include a parametriclight source such as an optical parametric oscillators (OPOs) and/or anoptical parametric amplifier (OPAs), which are typically tunable and cangenerate emission at wavelengths from the ultraviolet to themid-infrared range. OPOs and OPAs can have the benefit of good spectralproperty, high electrical-to-optical efficiency, ruggedness, and smallvolume.

In general, the fundamental or harmonics of Q-switched, diode-pumpedNd:YAG lasers serve as the pump for an OPO or OPA. For differentapplications, different wavelength can be generated. More specifically,300 nm radiations, which may be useful in Ozone detection and imaging,can be generated by sum frequency mixing of OPO radiation with theharmonics of the pump. 935 nm radiations, which can be used in watervapor measurements, can be directly generated by a 532-nm pumped OPO,with the optional use of injection seeding to further improve thespectral width close to Fourier limit.

The Focal Plane Array

The focal plane array 130 in the LIDAR system 100 shown in FIG. 1A candetect and/or record several types of information of the photons in thereturn pulse 15. In general, the focal plane array 130 may includesensing elements 132, such as Geiger-mode avalanche photodiodes (APDs),that can detect photons. In some cases, the sensing elements in thefocal plane array 130 may be sensitive enough to detect and time-stampsingle photons. The focal plane array 130 can also record the time offlight of the photons by, for example, comparing the arrival time of thephotons with the time at which the photons were transmitted. Inaddition, the focal plane array 130 can record the location (e.g.,sensing element position or region within the focal plane array 130) atwhich each photon is detected. The focal plane array 130 can also recordintensity information of the detected photons by, for example, recordingthe number of photons received at each pixel location or pixel region.

In practice, it can be helpful for the focal plane array 130 to have thefollowing properties: 1) high detection efficiency, i.e., highprobability that a photon is successfully detected every time it hitsthe detector; 2) low dark current, i.e., low probability that thedetector registers a photon when none is there; 3) low reset or “deadtime”, i.e., a short interval after a detection during which the devicecannot detect a new photon; 4) low cross-talk, i.e. low probability thatneighboring pixels detect photons arising from the detection process ina given pixel; and 5) low “timing jitter”, i.e., low uncertainty inspecifying when a photon arrives.

In one example, the focal plane array 130 can include an array of APDs,which are reverse-biased variants of p-n junction photodiodes.Typically, one pixel includes one APD, one biasing circuit, one timingcircuit, and an interface to the readout circuitry (e.g. shiftregisters) for the entire array. Without being bound any particulartheory or mode of operation, reversely biasing a p-n junction photodiodecan generate an electric field in the vicinity of the junction. Theelectrical field tends to keep electrons confined to the n side andholes confined to the p side of the junction. Absorption of a photonhaving sufficient energy (e.g., >1.1 eV for silicon) can produce anelectron-hole pair, after which the electron in the pair can drift tothe n side and the hole can drift to the p side, resulting in aphotocurrent flow in an external circuit.

The same principle can also allow an APD to detect light. However, anAPD is typically designed to support high electric fields so as tofacilitate impact ionization. More specifically, the electron and/or thehole in an electron-hole pair generated by photon absorption can beaccelerated by the high electric field, thereby acquiring sufficientenergy to generate a second electron-hole pair by colliding with thecrystal lattice of the detector material. This impact ionization canmultiply itself many times and create an “avalanche.” A competition candevelop between the rate at which electron-hole pairs are beinggenerated by impact ionization and the rate at which they exit thehigh-field region and are collected. The net result can be dependent onthe magnitude of the reverse-bias voltage: if the magnitude is below avalue (commonly known as the breakdown voltage), collection normallyoutruns the generation, causing the population of electrons and holes todecline. An APD operating in this condition is normally referred to as alinear mode APD. Each absorbed photon normally creates on average afinite number M (also referred to as the internal gain) of electron-holepairs. The internal gain M is typically tens or hundreds.

While M might be the average number of electron-hole pairs generated byone absorbed photon, the actual number may vary, inducing gainfluctuations. This gain fluctuation can produce excess noise, ormultiplication noise, which typically gets progressively worse with theincrease of M. Therefore, once the point is reached where themultiplication noise dominates over the noise introduced by downstreamcircuitry, further increases in gain may deteriorate the systemsignal-to-noise ratio. The multiplication noise can also depend onmaterial properties because, in general, electrons and holes can havedifferent capabilities to initiate impact ionizations. For example, inSi, electrons can be much more likely to impact ionize compared toholes. Therefore, it might be helpful for electrons to initiate impactionization in silicon-based APDs.

In another example, the focal plane array 130 can include an APDoperating in Geiger mode (also referred to as a Geiger-mode APD orGmAPD). A GmAPD operates when the reversely biased voltage is above thebreakdown voltage. In this case, electron-pair generation normallyoutruns the collection, causing the population of electrons and holes inthe high-field region and the associated photocurrent to growexponentially in time. The growth of photocurrent can continue for aslong as the bias voltage is above the breakdown voltage.

A series resistance in the diode, however, can limit the current growthby increasing the voltage drop across the series resistance (therebyreducing the voltage across the high-field region) as the current grows.This effect can therefore slow down the rate of growth of the avalanche.Ultimately, a steady-state condition can be reached in which the voltageacross the high-field region is reduced to the breakdown voltage, wherethe generation and extraction rates balance. Stated differently, theseries resistance can provide negative feedback that tends to stabilizethe current level against fluctuations. A downward fluctuation incurrent, for example, can cause a decrease in the voltage drop acrossthe series resistance and an equal increase in the drop across the APDhigh-field region, which in turn increases the impact-ionization ratesand causes the current to go back up.

The quenching circuit of the APD employed for the LIDAR system 100 canbe either passive or active. In a passive-quenching circuit, the APD ischarged up to some bias above breakdown and then left open circuited.The APD then discharges its own capacitance until it is no longer abovethe breakdown voltage, at which point the avalanche diminishes. Anactive-quenching circuit actively detects when the APD starts toself-discharge, and then quickly discharges it to below breakdown with ashunting switch. After sufficient time to quench the avalanche, theactive-quenching circuit then recharges the APD quickly by using aswitch. In LIDAR systems, where the APD typically detects only once perlaser pulse, the recharge time can be slow. There is also interest,however, in using the Geiger-mode APDs to count photons to measureoptical flux at low light levels. With a fast active-quenching circuit,the APD can be reset shortly after each detection (e.g., on a time scaleas short as nanoseconds), thereby allowing the APD to function as aphoton-counting device at much higher optical intensities.

In yet another example, the focal plane array 130 can include an arrayof superconducting nanowire single-photon detectors (SNSPDs), each ofwhich typically includes a superconducting nanowire with a rectangularcross section (e.g., about 5 nm by about 100 nm). The length istypically hundreds of micrometers, and the nanowire can be patterned incompact meander geometry so as to create a square or circular pixel withhigh detection efficiency. The nanowire can be made of, for example,niobium nitride (NbN), tungsten silicide (WSi), YBa₂Cu₃O_(7-δ), or anyother material known in the art.

In operation, the nanowire can be maintained below its superconductingcritical temperature T_(c) and direct current biased just below itscritical current. Without being bound by any particular theory of modeof operation, incident photons having sufficient energy to disrupthundreds of Cooper pairs in a superconductor can therefore form ahotspot in the nanowire. The hotspot itself typically is not largeenough to span the entire width of the nanowire. Therefore, the hotspotregion can force the supercurrent to flow around the resistive region.The local current density in the sidewalks can increase beyond thecritical current density and form a resistive barrier across the widthof the nanowire. The sudden increase in resistance from zero to a finitevalue generates a measurable output voltage pulse across the nanowire.

Various schemes can be employed in SNSPD to improve the detectionperformance. In one example, the SNSPD can employ a large area meanderstrategy, in which a nanowire meander is written typically across a 10μm×10 μm or 20 μm×20 μm area so as to improve the coupling efficiencybetween the incident photons and the SNSPD. In another example, theSNSPD can include a cavity and waveguide integrated design, in which ananowire meander can be embedded in an optical cavity so as to increasethe absorption efficiency. Similarly, a nanowire can be embedded in awaveguide so as to provide a long interaction length for incidentphotons and increase absorption efficiency. In yet another example,ultra-narrow nanowires (e.g., 20 nm or 30 nm) can be employed toconstruct the nanowire meander so as to increase the sensitivity tolow-energy photons.

In yet another example, the focal plane array 130 can include atransition edge sensor (TES), which is a type of cryogenic particledetector that exploits the strongly temperature-dependent resistance ofthe superconducting phase transition. In yet another example, the focalplane array 130 can include a scintillator counter which can detect andmeasure ionizing radiation by using the excitation effect of incidentradiation on a scintillator material, and detect the resultant lightpulses.

Embedded Processor

The embedded processor 140 in the LIDAR system 100 is operably coupledto the focal plane array 130 and processes raw data generated by thefocal plane array 130. The embedded processor 140 can be coupled to thefocal plane array 130 via a Readout Integrated Circuit (ROIC) (not shownin FIG. 1A). In operation, the ROIC can read out the photocurrentgenerated by the focal plane array 130, time stamp the photon arrivals,read out the pixel locations of the received photons, and convey theinformation off the ROIC and into the embedded processor 140.

In one example, the ROIC can store signal charge, generated by the focalplane array 130, at each pixel and then route the signal onto outputtaps for readout. The ROIC can accumulate signal charge on anintegrating capacitor. The total integrated charge can meet desireddynamic ranges by using appropriate capacitors and bias voltagesaccording to total charge=bias voltage×capacitance. The ROIC can storelarge signal charge at each pixel site and maintain signal-to-noiseratio (or dynamic range) as the signal is read out and digitized.

In another example, the ROIC can be digital as disclosed in U.S. Pat.Nos. 8,179,296; 8,692,176; and 8,605,853, each of which is incorporatedherein by reference in its entirety. More specifically, each pixel ofthe digital ROIC can include a full-dynamic-range analog-to-digitalconverter as well as local digital signal processing (DSP) support. Eachpixel can transfer data to any of its four nearest neighbors; the entirearray can be read out by using high-speed digital readout circuits. TheROIC can digitize the detector current within each pixel by incrementinga counter each time a small charge bucket is filled. In general, alarger detector current can result in a quicker filling of the bucketand quicker increment of the counter. In this ROIC, the total charge canbe given by the size of the charge bucket (in electrons) times thedigital value in the counter. In yet another example, the countercontaining the digital representation of the detector signal at eachpixel can be connected through a multiplexor to four nearest neighboringpixels. High-speed serializers located on the edge of the ROIC cantransfer the array contents onto a set of high-speed digital output tapsfor readout. Digital ROIC can have the benefit of being compatible withadvanced low-voltage (and thus low-power) deeply scaled IC fabricationprocesses that can allow increased circuit density and increased on-chipprocessing power. Furthermore, digital ROIC can have high speeds atwhich data can be read out.

The embedded processor 140, and optionally the ROIC, can be integratedwith the focal plane array 130 through several techniques. In oneexample, 3D integrated circuit technology can be employed. Morespecifically, a 3D focal plane array and circuits can be fabricated bytransferring and interconnecting fully fabricated silicon-on-insulator(SOI) substrates to a base wafer. The base wafer can be a high fillfactor detector wafer or another circuit wafer. Each functional sectioncan be referred to as a tier. Tier 2 can be transferred to the base tier(i.e., tier 1) via face-to-face infrared alignment, and oxide-oxidebonding. The handle silicon of a transferred tier can be removed bygrinding the silicon (e.g., to a thickness of ˜50 μm) followed by asilicon etching (e.g., in a 10% TMAH solution at 90° C.). Then 3D viascan be patterned to expose metal contacts in both tiers, followed byfilling the 3D vias with conductive material (e.g., damascene tungsten)to electrically connect the two tiers. A third tier, tier 3, can then beadded to the tier 1-2 assembly using the similar processes. Theresulting assembly can include an imaging tier (e.g., focal plane array130), a readout tier (e.g., the ROIC), and a signal processing tier(e.g., the embedded processor).

In another example, the assembly of the focal plane array 130 and theembedded processor 140 can be integrated horizontally. For example, thefocal plane array ROIC 130 and the embedded processor 140 can befabricated on the same substrate. Alternatively, the focal plane array130 and the embedded processor can be simply disposed adjacent to eachother on a common board as long as the coupling allows fast transmissionof data from the focal plane array 130 (or the ROIC) to the embeddedprocessor for prompt processing.

In yet another example, a hybrid technology including both 3D integratedcircuit and horizontal integration. More specifically, the focal planearray 130 can be integrated with the ROIC via 3D integration circuittechnology, while the embedded processor is disposed adjacent to thefocal plane array 130.

The embedded processor 140 can be realized either as a FieldProgrammable Gate Array (FPGA) or an Application Specific IntegrationCircuit (ASIC). In one example, the FPGA approach can be employed forits relatively simple design. A Field-Programmable Gate Array (FPGA) isgenerally a semiconductor device containing programmable logiccomponents conventionally referred to as “logic blocks” and programmableinterconnects. Logic blocks can be programmed to perform the function ofbasic logic gates such as AND, and XOR, or more complex combinationalfunctions such as decoders or mathematical functions. For example, aVIRTEX-7 FPGA, manufactured by XILINX, can deliver 2 million logiccells, 85 Mb block RAM, and 3,600 DSP48E1 slices for new possibilitiesof integration and performance.

In another example, the ASIC approach can be employed for its morepowerful computation capability. An ASIC is generally an integratedcircuit designed for a particular use, rather than intended forgeneral-purpose use. Example ASICs include processors, RAM, and ROM.Therefore, ASIC can be more complicated, and thus more expensive, thanFPGA. In practice, the decision of using a FPGA or an ASIC may depend onseveral factors, such as the budget, time constraint, and fabricationcapabilities.

Second Processor

The second processor 150 in the LIDAR system 100 shown in FIG. 1A isconfigured to further process the data provided by the embeddedprocessor 140 and ultimately generate a 3D image of the scene 20. Thesecond processor 150 is generally more powerful than the embeddedprocessor 140 and can perform functions such as 3D registration.

In one example, The second processor 150 can be a microprocessor,microcontroller, CPU, or suitable FPGA. In another example, the secondprocessor 150 may be any electronic device that can analyze and/orprocess data and generate one or more 3D images. To this end, the secondprocessor 150 may include or be associated with a computing device, suchas a portable computer, personal computer, general purpose computer,server, tablet device, a personal digital assistant (PDA), smart phone,cellular radio telephone, mobile computing device, touch-screen device,touchpad device, and the like.

The data link 160 that convey data between the embedded processor 140and the second processor 150 may be any suitable wired and/or wirelesscommunication interfaces by which information may be exchanged. Examplewired communication interfaces may include, but are not limited to, USBports, RS232 connectors, RJ45 connectors, and Ethernet connectors.Example wireless communication interfaces may include, but are notlimited to, Bluetooth technology, Wi-Fi, Wi-Max, IEEE 802.11 technology,radio frequency (RF), LAN, WAN, Internet, shared wireless accessprotocol (SWAP), Infrared Data Association (IrDA) compatible protocolsand other types of wireless networking protocols, and any combinationsthereof.

FIG. 1B is a block diagram showing a general architecture of the LIDARsystem 100 in FIG. 1A. The general architecture includes three parts.The first part is a components part 101 which include large, fastGeiger-mode focal plane arrays 131 and a powerful embedded processor140.

The second part is an architecture part 102 which includes functionsthat can be performed by the components part. More specifically, theGeiger-mode focal plane arrays 131 can allow snapshot imaging 180 ofmultiple portions of a scene. The snapshot imaging 180 can further allow3D registration 184 via, for example, sequential image registration. TheGeiger-mode focal plane array 131 can also allow simple image formation182 such as image formation for each portion of the scene. The embeddedprocessor 140 can perform image formation from raw data acquired by thefocal plane array. The embedded processor 140 can also remove noise andredundancy (186) from the raw data.

The third part is a system impact part 103, which includes advantagesthat can be derived from the components and architecture parts. Forexample, the 3D registration 184 can allow the use of simple,lightweight pointing and scanning systems 190, instead of the bulkyprecision opto-mechanical scanners, thereby reducing the SWaP of theresulting system 100. In addition, noise and redundancy removal 186 bythe embedded processor 140 can reduce the data bandwidth for storage ordownlink (data link 160) such that the weight and power consumption ofthe resulting system 100 can be decreased.

FIG. 1C shows an example LIDAR system including an FPGA as an embeddedprocessor. The LIDAR system 101 shown in FIG. 1C includes a light source111, which further includes a power supply 112, a pair of pumping laserdiodes 113 a and 113 b, a pair of microchip lasers 116 a and 116 b, abeam combining and shaping optic 117, and a transmission window 115. Thepair of pumping diodes 113 a and 113 b can deliver pumping beams at 808nm to pump the pair of microchip lasers 116 a and 116 b, which canoperate at 1064 nm. The pump diodes 113 a and 113 b could be driven in aCW mode, resulting in a burst of short (sub-nanosecond) pulses producedby the microchip lasers 116 a and 116 b. In one example, the burst ofshort pulses can have a duration (e.g., on the order of milliseconds)matching the duration of the electrical pumping delivered to the pumplaser diodes 113 a and 113 b. In another example, the pump diodes 113 aand 113 b can be controlled with microsecond-scale timing accuracy,thereby allowing the microchip laser pulse repetition frequency to bemore tightly controlled. In yet another example, the microchip lasers116 a and 116 b can be pumped continuously. Switching circuitry thataccommodates millisecond-scale timing can be small and efficient,whereas microsecond-scale control typically uses bulkier andless-efficient circuitry.

The output beams from the pair of microchip lasers 116 a and 116 b canbe combined and shaped by the beam combining and shaping optic 117. Inone example, the beam divergence (in each of two angular directions) canbe configured to match the field-of-view of the detector (e.g., focalplane array 131) in the receiver section. In another example, the beamdivergence can be smaller, but steerable, so as to illuminate a desiredsub-region on the detector array 131. The transmission window 115 canthen transmit the beam while maintaining a hermetic seal 181 againstcontaminants. The transmission window can be coated (e.g., thin films)to block stray pump light. The transmitter section of the LIDAR system101 can also contain a means to sample the outgoing pulses for timereferencing and perhaps pulse energy monitoring.

The LIDAR system 101 also includes focal plane array 131 operablycoupled to support circuitry 142 and a FPGA die 143. The FPGA die 143can be used for on-board processing as described in detail below. Thefocal plane array 131 can receive light transmitted by the light source111 and reflected/scattered by the scene through a receiving window 135,which maintains a hermetic seal 181 against contamination. The windowcan be coated to block ambient light at wavelengths other than thetransmitter wavelength.

LIDAR Techniques Including On-Board Data Processing

FIG. 2A illustrates a LIDAR technique that can allow reduction of thetotal SWaP of LIDAR systems by using on-board data processing.Photon-counting LIDAR (also referred to as single-photon LIDAR) makesefficient use of each received photon, and thus can be an attractiveoption for compact systems. However, since the time and location of eachdetected photon is recorded in photon-counting LIDAR, the data outputrates of large focal plane arrays can be very large. Therefore, theinfrastructure to transmit data from the detector to processor for 3Dimage generation can be a dominant fraction of the total SWaP of aphoton-counting LIDAR system. On-board processing can reduce thebandwidth required to transfer data off the chip, thereby reducing theweight of the data transmission infrastructure as well as the powerconsumption to transmit the data.

The method 200 shown in FIG. 2A can reduce the data rate by a factor ofabout 10 to about 100, thereby substantially reducing system power,weight, and volume. The method 200 starts from step 210, in which afirst portion of a scene is illuminated with at least one light pulse.At step 220, at least one raw point cloud is acquired by detecting andrecording photons reflected or scattered from the first portion of thescene. A focal plane array (e.g., focal plane array 130 shown in FIG. 1and discussed above) can be used to acquire this raw point cloud.

The raw point cloud can include several types of data. For example, theraw point cloud can include data points that are generated by photons inthe light pulse and represent the distance between the focal plane arrayand the portion of the scene illuminated by the light pulse. However,the raw point cloud may also include noise data points and/or redundantdata points. The noise data points may originate from dark currentgenerated by thermally initiated electron-hole pairs (rather than fromthe absorption of an incident photon). The noise data points may alsooriginate from reflected photons that are not in the light pulsestransmitted by the LIDAR system. For example, the scene can reflect orscatter the sunlight or artificial lighting to the focal plane array,thereby generating data points that typically do not include thedistance information useful for 3D imaging. In addition, as discussedabove, an avalanche photodiode (APD), when employed as the focal planearray, may also have multiplication noise due to the variation of gain.

The redundant data points are typically generated by photons in thelight pulse, but they can be cumulative. In principle, only one photonis needed to derive the distance between the focal plane array and thepoint which reflects the photon. In practice, on average 10-20 photonsare collected for each desired measurement in order to be able todiscriminate against background noise and to have high confidence(P_(det)>0.995) of detecting any real surfaces present in the scene. Therange and intensity (2 numbers) can be derived from the timinginformation of these 20 photon detection events (20 numbers).

At step 230, a first processor removes at least one noise data pointand/or at least one redundant data point from the raw point cloud so asto generate a processed point cloud having fewer data points that theraw point cloud. The first processor can be substantially similar to theembedded processor 140 shown in FIG. 1 and described in detail above.Since the first processor can be integrated with the focal plane arrayor at least physically close to the focal plane array (e.g., less than10 inches, 5 inches, 2 inches, or 1 inch), data can be transmittedbetween the focal plane array and the first processor promptly withoutsophisticated hardware.

At step 240, the processed point cloud is conveyed to a second processorfor further processing including, for example, 3D image registration.The second processor can be substantially similar to the secondprocessor 150 shown in FIG. 1 and described in detail above. Since theprocessed point cloud typically has a much smaller number of data pointscompared to the raw point cloud, infrastructure of data transmissionbetween the first processor and the second processor can be much lesscomplicated (and less power consuming), thereby decreasing the SWaP ofthe entire system.

In one example, the first processor processes the raw point cloudgenerated by each light pulse at step 230 and then conveys the processedpoint cloud to the second processor at step 240. In another example, thefirst processor collects a plurality of raw point clouds generated by aplurality of light pulses and then processes the plurality of raw pointclouds simultaneously before conveying the processed point cloud to thesecond processor at step 240. In yet another example, the firstprocessor can preliminarily process (e.g., remove noise from) each rawpoint cloud generated by each light pulse but then store the processedpoint cloud. The first processor can then perform a second processingstep to the stored point clouds so as to generate the processed pointcloud and convey the processed point cloud to the second processor.

The first processor can process the raw point cloud at step 230 invarious ways. In one example, the first processor removes noise datapoints from the raw point clouds. As discussed above, noise may comefrom dark current in the focal plane array, photons from sources otherthan the light pulses (e.g., the sun). These noise data points can havea common feature in that they are typically distributed in timesubstantially uniformly. Since thermally initiated electron-hole pairscan arise at any moment (e.g., randomly), the detected signal from thesethermal electron-hole pairs are accordingly random in time(statistically, the data points can spread uniformly in temporaldomain). In contrast, signal data points of interest, which aregenerated by photons that undergo a round trip between the LIDAR systemand scene, are concentrated in time corresponding to the range to thesurface which reflects the illumination light back to the receiver.Therefore, the first processor can filter and remove the noise datapoints by taking advantage of this difference in temporal distributionsbetween signal data points and noise data points.

In another example, the first processor can process the raw point cloudat step 230 by removing redundant data points. As introduced above,redundant data points may come from photons reflected by the same pointin the scene. In general, the first processor may find redundant datapoints by examining the distance information in data points thatcorrespond to a common point in the scene. For a raw point cloudgenerated by one light pulse, the first processor may find redundantdata points by examining the distance information contained in datapoints collected at the same pixel location. For raw point cloudsgenerated by more than one light pulse, image alignment can be helpfulin facilitating redundancy removal.

In yet another example, the first processor can implement coincidenceprocessing steps that can include both noise removal and redundancyremoval. Coincidence Processing (CP) generally inputs 3D point cloudsgenerated by detection of single photons, and output estimates of theunderlying 3D geometry of a scene, while also reducing noise content. CPcan use a non-linear process to reject the noise points and estimate thegeometry (and sometimes reflectivity) of the scene using the positionand amplitude of these signal concentrations. Advanced CP can addressmany system non-idealities such as 3D point-spread function, detectorblocking, detector saturation, detector responsivity and intrinsic darkcount rate, relative illumination intensity, and crosstalk across thearray, among others.

Typically the coincidence processing can be accomplished in a 3D griddedspace. Voxels with a number of detections attributed to them can bestatistically more significant compared to the expected background rate.Such statistical difference can result in an output point in the outputpoint cloud. In one implementation of CP, the voxel size can be smallerthan the 3D point spread function (PSF) of the hardware. Each detectionevent can be attributed to one or multiple voxels, weighted by the PSF.In one implementation, the significance of a detection can be tabulatedbased upon the known pixel responsivity and relative illumination ofthat pixel. An expected background rate can be attributed to the voxelas informed by the pixel dark count rate (DCR) and number of times thevoxel was interrogated by a pixel that was sensitive.

For example, the first processor can perform a coincidence processingstep as following: 1) for a given scene point, identify the set ofrecorded ranges from one or more light pulses that are reflected by thisscene point; 2) examine the recorded ranges and estimate the mostprobable range; and 3) assign the most probable range to the scene pointfor 3D image generation. Estimating the most probable range from theplurality of recorded ranges can be based on, for example, statisticalanalysis.

In yet another example, the first processor executes one or moreestimation processes, based on the available measurements, to estimatethe location and properties of surfaces in the scene. Properties thatmay be estimated can include size, reflectivity, pose, polarizationcharacteristics, and motion (speed and vibration). More details can befound in, for example, “Laser Radar: Progress and Opportunities inActive Electro-Optical Sensing” (ISBN 978-0-309-30216-6, from theNational Academies of Science), the entirety of which is incorporated byreference herein.

In yet another example, when multiple illumination wavelengths are used,the first processor can conduct spectral analysis of each surface tocategorize the data points from each wavelength. In yet another example,the first processor may also detect and track an object, which might bebehind partial occlusions from time to time, moving through the scene in3D.

The method 200 illustrated in FIG. 2A can reduce the SWaP of a LIDARsystem by reducing the number of data points that are transmitted forprocessing. More specifically, the first processor can ingest the photontiming information accumulated from each of many pixel exposures. Apixel exposure can be one pixel “looking at” the scene for one laserpulse. Typically 10²-10⁴ pixel exposures per spatial resolution elementare used in photon-counting LIDAR systems. The actual number of pixelexposures may depend on complexity of the scene. In one example, thepixel exposures can be achieved by accumulating output from a singlepixel during many laser illumination pulses. In another example, thepixel exposures can be achieved by accumulating outputs from manydistinct pixels, each of which is pointed within the same resolutionelement in the scene and illuminated by just one laser pulse. In yetanother example, the pixel exposures can be achieved by a combination ofthese approaches.

The data rate output from the first processor is typically much lessthan the raw data rate generated by the focal plane array. For example,a typical photon counting LIDAR interrogating a simple scene (e.g., aflat ground) can be operated such that on average 10-20 photons aredetected in each resolution element. The output from the first processorcan be a single range value representing the distance between theresolution element and the focal plane array. Therefore, one data pointcan replace the 10-20 raw data points, thereby achieving a data ratereduction factor of about 10 to about 20.

The data rate reduction rate can be even larger in heavily foliatedscenes, which can be of particular interest in, for example, LIDAR-basedland surveys. Assuming a canopy obscuration fraction of 90%, thedetector may receive 90% of the returned photons from the canopy andonly 10% from the ground area of interest. In addition, the canopy andthe ground underneath the canopy can have different reflectivities. Ifthe ratio of the reflectivity of the ground surface to the canopy is1:4, then there can be 40 times fewer photons from the ground than fromthe canopy. To achieve the desired 10 detections from the ground, thesystem may record 400 detections from the canopy. These 410 detectionscan be summarized into only a few surface height estimates by the firstprocessor, thereby achieving a data rate reduction rate of about 100.High reduction or compression can also be achieved in the presence ofhigh background illumination.

FIG. 2B shows a raw point cloud acquired, for example, at step 220 ofthe method 200. FIG. 2C shows a processed point cloud obtained, forexample, at step 240 of the method 200. The processing can remove noiseand redundancy as well as increase the sharpness of the point cloud.

FIG. 3 illustrates a LIDAR method 300 which can be helpful in surveyingcomplex scenes such as forest areas. At step 310 of the method 300, aportion of the scene is illuminated by a light pulse. At step 320, a rawpoint cloud is acquired by detecting and recording photons reflected orscattered from the illuminated portion of the scene. Then the method 300determines whether sufficient exposure (i.e., sufficient number ofphoton received) of the portion of the scene is acquired at step 330.Whether sufficient exposure is acquired may depend on several factors,such as the energy (and thus the number of photons) in the light pulse,the complexity of the scene (e.g., whether heavy canopy exists), thereflectivity of the surface of interest, the atmospheric transmission ofthe photons (e.g., the atmosphere may absorb more photons at onewavelength than another), and the sensitivity of the focal plane arrayto the photons in the light pulse, among others.

If it is determined that sufficient exposure is acquired, the method 300proceeds to step 340, in which a first processor processes the raw pointclouds by, for example, removing noise and redundancy and performingcoincidence processing so as to generate a processed point cloud. If,however, it is determined that the acquired exposure is not sufficient,the method 300 returns to step 310 and illuminate the portion of thescene by another light pulse. This loop may continue until the exposureis deemed to be sufficient. At step 350, the processed point cloud fromone or more raw point clouds is conveyed to a second processor forfurther processing including, for example, 3D image registration.

LIDAR Techniques Using Light Bursts and Sequential Registration

FIG. 4 shows a LIDAR technique which can reduce the total SWaP of aLIDAR system by replacing the otherwise bulky precision opto-mechanicalscanner and IMU with sequential 3D registration of the imagery. Insteadof registering point clouds with respect to a point whose location isknown or can be estimated with high accuracy, such as the scanner, theprocess 400 shown in FIG. 4 involves registering the point clouds toeach other based on overlap among the point clouds. This allowselimination of the opto-mechanical scanner from the LIDAR system. Inaddition, the bulky IMU can either be removed or replaced by a smaller,lower-performance IMU so that there is still some approximateorientation knowledge.

At step 410 of the method 400, a first portion of a scene is illuminatedby a first light burst, which includes a first plurality of lightpulses. The total duration of the first light burst is substantiallyequal to or less than 1 millisecond. At step 420, a first plurality ofpoint clouds are generated. Each point cloud in the first plurality ofpoint cloud corresponds to each light pulse in the first plurality oflight pulses. The first plurality of point clouds is generated bydetecting photons reflected or scattered from the first portion of thescene using a focal plane array. A first data point in the firstplurality of point clouds represents a first distance between the focalplane array and a first scene point in the first portion of the scene.At step 430, a first 3D representation of the first portion of the sceneis estimated based at least in part on the first plurality of pointclouds.

At step 415 of the method 400, a second portion of a scene isilluminated by a second light burst, which includes a second pluralityof light pulses. The second portion of the scene is at least partiallyoverlapping with the first portion of the scene. The total duration ofthe second light burst is substantially equal to or less than 1millisecond. At step 425, a second plurality of point clouds aregenerated. Each point cloud in the second plurality of point cloudcorresponds to each light pulse in the second plurality of light pulses.The second plurality of point clouds is generated by detecting photonsreflected or scattered from the second portion of the scene using thefocal plane array as used in step 420. A second data point in the secondplurality of point clouds represents a first distance between the focalplane array and a second scene point in the second portion of the scene.At step 435, a second 3D representation of the second portion of thescene is estimated based at least in part on the second plurality ofpoint clouds.

At step 440 of the method 400, a 3D image of the scene is generatedbased at least in part on the first 3D representation of the firstportion of the scene and the second 3D representation of the secondportion of the scene.

Steps 410-430 and steps 415-435 can be practiced in various orders. Inone example, steps 415-435 can be performed after the completion of thesteps 410-430. In other words, the method 400 estimates the first 3Drepresentation of the first portion of the scene before initiating theestimation of the second 3D representation of the second portion of thescene.

In another example, the method 400 can perform the steps 410 and 420 toacquire the first plurality of point clouds, followed by the performanceof steps 415 and 425 to acquire the second plurality of point clouds.Then the method 400 can group process the first plurality of pointclouds to generate the first 3D representation and the second pluralityof point clouds to generate the second 3D representation.

Although FIG. 4 shows only two estimations of 3D representations of twoportions of a scene, in practice, the number of 3D representations tocreate a complete 3D image of an extended scene can be more than two. Ifthere are more than two 3D representation involved, the 3D imagegeneration can also be performed in various ways. In one example, step440 can take the first two 3D representations that are available for 3Dimage generation, while the estimation of more 3D representations isstill under progress. In another example, the method 400 can collect allthe 3D representations first before generating the 3D image. In yetanother example, the method 400 can collect all the point clouds firstbefore estimating any 3D representations, after which the method 400proceeds to the generation of the 3D image.

The number of light pulses contained in the first light burst and/or thesecond light burst can be dependent on several factors, including, theenergy (and thus the number of photons) in each light pulse, thecomplexity of the scene (e.g., whether heavy canopy exists), thereflectivity of the surface of interest, the atmospheric transmission ofthe photons (e.g., the atmosphere may absorb more photons at onewavelength than another), and the sensitivity of the focal plane arrayto the photons in the light pulse, among others. The number of lightpulses can influence the resolution of the resulting 3D image of thescene. In general, larger number of the light pulses in each burst canlead to a higher resolution. In one example, each light burst caninclude about 2 to about 1000 pulses. In another example, each lightburst can include about 5 to about 500 light pulses. In yet anotherexample, each light burst can include about 20 to about 50 light pulses.

The first light burst and the second light burst can include differentnumbers of light pulses. For example, the first portion of the scenemight be more complex or have a different surface than the secondportion of the scene. Therefore, the first light burst may contain 20light pulses while the second light burst contains only 5 light pulses.

The pulse duration of each light pulse in the light bursts can be shortso as to allow a light burst to contain a large number of light pulses.In one example, each light pulse has pulse duration less than 5 ns. Inanother example, each light pulse has pulse duration less than 1 ns. Inyet another example, each light pulse has pulse duration less than 500ps, 200 ps, or 100 ps. In yet another example, femtosecond light pulses(pulse duration <1 ps) can be employed.

The focal plane array used in steps 420 and 425 can be substantially thesame focal plane array 130 shown in FIG. 1 and described in detailabove. More specifically, the focal plane array can include, forexample, an avalanche photodiode (APD), a Geiger-mode APD, asuperconducting nanowire single-photon detector (SNSPD), a transitionedge sensor (TES), a scintillator counter, a charge-coupled device (CCD)or complementary metal-oxide semiconductor (CMOS) array, among others.

The focal plane array can also include an array of Geiger-mode APDs orother single-photon detectors. In one example, the focal plane array caninclude an array of APDs including at least 16 rows and 16 columns. Inanother example, the array can include an array of APDs including atleast 64 rows and 64 columns. In yet another example, the array caninclude an array of APDs including at least 128 rows and 128 columns.

The data points in the first plurality of point clouds and the secondplurality of point clouds can include the time of flight information ofthe detected photons so as to derive the distance between the focalplane array and the respective portion of the scene. The data points canalso include location information of the pixels at which the photons arereceived. This location information can be helpful in subsequentestimation of 3D representations and 3D image generation.

In one example, the first 3D representation and the second 3Drepresentation can be generated via the embedded processor 140 shown inFIG. 1 and described in detail above. In this case, estimation of 3Drepresentation can be performed on-board, thereby further reducing thetotal SWaP of the resulting LIDAR systems. In another example, the 3Drepresentations can be generated after the collection of all pointclouds by an off-board processor. The 3D representations can begenerated using the coincidence processing technique described above. Inaddition, the estimation can be performed in an angle-angle rangecoordinate frame (i.e. aligned to optical rays traced outward toward thescene from each pixel in the focal plane array), without the necessityto calculate the Cartesian coordinates as typically required in LIDARtechniques.

In one example, the 3D representations are processed point clouds. Eachdata point in the point cloud represents a location of a point in thescene and the distance between the point and the focal plane array. Inanother example, the 3D representation can be 3D images of therespective portion of the scene. Data points in the 3D representationscan represent the location of the points in the scene and the surfaceproperties of the points. The surface properties may include, forexample, elevation of the point and type of the point (ground, canopy,ice, asphalt, etc.).

The first portion of the scene illuminated at step 410 is at leastpartially overlapping with the second portion of the scene illuminatedat step 415 so as to allow sequential registration of images. In oneexample, the first portion of the scene is overlapping with the secondportion of the scene by about 30% to about 60%. In another example, theoverlapping percentage can be at least 50%. In yet another example, theoverlapping percentage can be at least 70%. The overlapping percentagecan be dependent on, for example, the number of resolution elements(i.e., element in the scene mapped to one pixel in the detector) in thescene. In general, a larger number of resolution elements in a portionof the scene can result in lower overlapping percentage requirement.

In one example, the first plurality of point clouds are taken when thefocal plane array is at a first location, while the second point cloudsare taken when the focal plane array is at a second location differentfrom the first location. For example, in airborne LIDAR, the plane maysurvey a scene along a straight line, and each plurality of point cloudscan be taken at a different location along the straight line. In anotherexample, the first plurality of point clouds are taken when the focalplane array is at a first orientation, while the second point clouds aretaken when the focal plane array is at a second orientation differentfrom the first location. For example, in a ground-based LIDAR, the focalplane array can be tilted toward different portions of a scene so as tocover a wide area. In this case, each plurality of point clouds can betaken when the focal plane array is at a different orientation. In yetanother example, both the location and the orientation of the focalplane array can be different. For example, in an airborne LIDAR, theplane may fly around a tree to take point clouds at different anglestoward the tree so as to collect more photons reflected from the groundunderneath the tree. In this case, each plurality of point cloud istaken while the focal plane array is at a different location (e.g.,location along a periphery above the tree) and a different orientation(e.g., tilting).

The 3D image generation at step 440 can employ the sequential imageregistration technique since adjacent portions of the scene areoverlapping. The sequential image registration can compare twoindividual 3D representations and find out the overlapping region, afterwhich the two individual 3D representations can be aligned together andform an image of a larger portion, i.e., the sum of the first portionand the second portion. The process can continue until the entire sceneof interest is covered. Sequential image registration can take advantageof the relative location of each data point in the point clouds, withoutthe necessity to know the exact and absolute geolocation of the eachdata point. Therefore, the bulky precision opto-mechanical scanner andthe IMU, which are conventionally employed to derive the geolocation ofthe data points, can be eliminated. The total SWaP of the resultingLIDAR system can be accordingly reduced. At step 440, coincidenceprocessing can also be optionally performed in the overlapping regionbetween the two or more individual 3D representations so as to furtherreduce noise/redundancy and improve accuracy.

Although exact geolocation of each data point in the point clouds is notnecessary to produce the final 3D image, geolocation information canstill be derived so as to allow the georeferencing of the 3D image. Inone example, one point in the 3D image can be georeferenced using a GPS,for example, at the beginning or end of the data acquisition process.The rest of the points in the 3D image can be derived based on thisgeoreferenced point since the relative location between the rest of thepoints and this georeferenced point is known. In this case, the entire3D image can still be georeferenced without the bulky opto-mechanicalscanner and the IMU.

FIG. 5 illustrates a method of imaging a wide area scene. The method 500starts by illuminating a portion of a scene using a light burst at step510, followed by acquiring, at step 520, a plurality of point cloudscorresponding to the plurality of light pulses contained in the lightburst. At step 530, the plurality of point clouds is processed so as toestimate a 3D representation of the portion that was illuminated at step510. At step 540, the method 500 determines whether sufficient coveragehas been made with respect to the scene, i.e., whether sufficient areaof the scene has been imaged. If not, the method proceeds to step 560 atwhich the LIDAR system is moved to a different portion of the scene andrepeats steps 510 to 540 until sufficient coverage is completed. Movingto a different portion can be achieved by, for example, moving the focalplane array to a different location, a different orientation, or both.If the coverage is deemed to be sufficient at step 540, the methodproceeds to step 550 to generate the 3D image of the scene using thegenerated 3D representations.

At least due to the reduced SWaP, the entire LIDAR system implementingthe methods illustrated in FIGS. 4 and 5 can be as small as the size ofa child's thumb. More specifically, the physical device can be a smallpackage (e.g., 10 cc, but could be larger or smaller) containing amicrochip laser, a focal plane array of avalanche photodiodes operatingin Geiger-mode (GmAPD FPA), and a computing device such as a fieldprogrammable gate array (FPGA), digital signal processor (DSP), orapplication specific integrated circuit (ASIC). One window in thephysical device can transmit the laser radiation out, and the otherwindow, ordinarily placed behind a focusing optic, transmits the returnsignal to the detector array. The device operates in a burst mode, inwhich a sequence of laser pulses illuminates the scene at a pulserepetition frequency (PRF) consistent with efficient laser operation andachievable readout rates of the detector array. A typical PRF can be,for example, about 10 kHz to about 200 kHz, or about 20 kHz to about 100kHz, or about 30 kHz. A typical burst length can be, for example, 1msec. The burst length can be sufficiently short so that elements in theimage move less than a significant fraction of a pixel during the burst.

The detector array records pixel firing times (and therefore round-triplight travel times, and therefore range) for each pulse. This timinginformation and the resulting range information can be aggregatedtogether in a processing element over the duration of the burst. Oncethe burst is complete, the laser and detector array can be transitionedto a low-power standby mode until the next burst is fired.

During this inter-burst interval the processor can remove redundancy andnoise so the data volume for the burst is reduced by a factor of 10¹-10²depending on the scene type and fidelity requirements. The data for theburst can be transferred to an output buffer and streamed off the deviceduring the next burst and processing interval. The burst can be issued,notionally, after the sensor field-of-view (FOV) has moved a significantfraction of the FOV. Accelerometers or other sensors can be employed tomeasure this movement. Once the imagery data is offloaded from thesensor (and perhaps the platform), it can be aligned to images frompreceding (and perhaps following) bursts using 3D registrationtechniques. Taken together, many hundreds of bursts can form a clean andusefully-large image.

The above approach can be of particular significance in applicationswhere sensor SWaP is constraint. To make 3D registration succeed, thescene imaged in a single burst can contain many features (resolutionelements) with characteristic sizes on the order of the desired dataresolution. This may help determine a lower bound on the size of theindividual image chips. For example, if 30 spatial resolution elementsare desired in each direction, a total of approximately 10³ resolutionelements are desired in the image chip. Given an assumption aboutangular slew rates or platform speeds, there can be also an upper boundon the inter-burst period so as to allow at least partial overlappingbetween sequential image chips.

LIDAR Techniques with Configurable Area of Interest

FIG. 6 illustrates a LIDAR method which can be efficient in situationswhen the image quality requirements may vary dramatically across thearray field of view (FOV). The method can also be useful when the targetscene or object is relatively small, or the size of the target scene orobject is unknown before the measurement.

The method 600 shown in FIG. 6 starts from step 610, at which a scene isilluminated by an illumination light pulse, followed by acquiring, atstep 620, an illumination point cloud from photons in the illuminationlight pulse reflected or scattered from the scene using a focal planearray. At step 630, an area of interest in the focal plane array isdetermined based at least in part on the illumination point cloud. Thearea of interest can be a subset of the entire focal plane array. Atstep 640, the scene is illuminated by a plurality of signal lightpulses. For each signal light pulse in the plurality of signal lightpulses, a respective signal point cloud is acquired by reading out datafrom the area of interest in the focal plane array, as performed at step650. At step 660, a 3D image of at least a portion of the scene isgenerated based at least in part on a plurality of signal point cloudsfrom the plurality of signal pulses acquired in step 650.

In one example, the divergences of the illumination light pulse and thesignal light pulse can be the same and delivered by the same lightsource (e.g., the light source 110 show in FIG. 1 and described indetail above). In another example, the illumination light pulse and thesignal light pulse can have different divergences. For example, theillumination light pulse can have a larger divergence such that thereflected light can illuminate the entire focal plane array. The signalpulse, however, can have a smaller divergence only to cover the area ofinterest in the focal plane array determined by the illumination lightpulse. The signal pulses may be steered to a region of interest asdirected by a system controller, based in part on initial analysis ofthe illumination point cloud or cues from other sensors. Photondetections that are useful for creating a 3D image may only come fromthe region of the entire focal plane array that is illuminated by thesignal light pulse. For the illuminated region, data can be read out ina manner that preserves the fine timing detail. The remainder of thearray can also be read out, but without the high precision timinginformation, so the electrical power and data communication bandwidthcan be reduced.

The readout region of interest can occupy different percentages of areasin the focal plane array. In one example, the area of interest is about1% to about 10% of the area of the focal plane array. In anotherexample, the area of interest can be less than 1% of the area of thefocal plane array. In yet another example, the area of interest can beless than 20% of the area of the focal plane array.

The number of signal pulses in the plurality of signal pulses can bedependent on the desired exposure of the scene, or the desiredresolution, or the scene complexity (e.g. the presence of heavyfoliage). In general, a larger number of signal pulses illuminating thescene can lead to a higher resolution of the resulting images. In oneexample, the plurality of signal light pulses comprises at least 5signal light pulses. In another example, the plurality of signal lightpulses comprises about 5 to about 20 signal light pulses. In yet anotherexample, the plurality of signal light pulses comprises about 5 to about100 signal light pulses.

The region of interest to be read out in the high-precision timing modecan be determined by, for example, counting a number of photons receivedat each pixel or group of pixels in the focal plane array and thendetermining the area of interest based at least in part on the number ofphotons at each pixel or group of pixels in the focal plane array. Ingeneral, a higher density of photons concentrated in a region tends toindicate that the region was illuminated by the signal light pulse andis an area of interest. The number of photons can be counted within apredetermined time window, such as a time window within which allphotons reflected by the scene are expected to return to the focal planearray.

Although FIG. 6 shows only one 610 illumination pulse, in practice, thenumber of 610 illumination pulses employed to determine the area ofinterest can be more than one. For example, a second illumination pulsecan be helpful when the first image generated by the first illuminationpulse does not capture the target scene sufficiently (e.g., because thefirst illumination pulse misses the target or because the signal levelis low).

In addition, the area of interest in the focal plane may change duringthe data acquisition in imaging the scene. For example, the focal planearray may be disturbed and dislocated from its original location ororientation. In this case, it can be helpful to re-determine the area ofinterest. In another example, the focal plane array can be mounted on anairborne platform which scans a scene by flying over the scene. In thiscase, it can be helpful to periodically check the area of interestduring the flight so as to update the area of interest.

In yet another example, two illumination pulses of different sizes(divergences) can be employed to the illuminate the scene and determinethe area of interest. For example, a small beam can be used to generatea large number of photo-detections (e.g., 10-50) for each of a number ofsmall pixels, resulting in high resolution 3D imagery, while a largebeam can illuminate a larger area, but collect enough photon-detectionsonly when aggregated into coarser resolution (e.g., larger) pixels.These coarse images can be used for coarse range determination andtarget acquisition. The wide and narrow beams can be multiplexed (e.g.,one burst with the wide beam, several bursts with the narrow beam) todetermine the next area of interest to illuminate.

LIDAR techniques with configurable area of interest can reduce the totalSWaP of a LIDAR system by eliminating the bulky opto-mechanical scannersand IMU, while simultaneously limiting the sensor data volume. Morespecifically, active imaging systems such as LIDAR expend powerilluminating, sensing, and processing imagery. In general, differentparts of a large scene can have different reflectivities and differentscene complexity. Therefore, it generally takes different numbers ofinterrogations to achieve the desired imaging performance for thesedifferent parts. To reduce power usage, it can be helpful to limit theintegration time devoted to each region of a scene while achieving thedesired product quality. The desired product quality may take account offactors such as scene complexity and spatial resolution. For example,less integration time is typically needed to form a useful image of aflat grass field than to detect objects under heavy canopy. Theintegration time per unit area can depend on, for example, thereflectance of the surface, the obscuration fraction, the desiredspatial resolution, and the number of surfaces present within a spatialresolution element, and the acceptable rate of false positives and falsenegatives output from the signal processing chain.

Additionally, in some applications where the LIDAR is cued by anothersensor, normally only a small sub-region of a scene is of any interest.In general, the desired integration time per unit area can be a functionof position within the addressable sensor field of regard (FOR). To makeefficient use of the available laser power and detector pixels, typicalLIDAR systems use a precision pointing and scanning system to direct anarrow sensor field of view to the intended region of the sensor'scomplete FOR. As introduced above, this precision pointing systemtypically comprises a significant fraction of system SWaP.

In contrast, the LIDAR method shown in FIG. 6 and the physical systemimplementing the method can eliminate the precision pointing systems,thereby reducing the SWaP. The LIDAR system can include a pulsed lightsource (e.g., a laser) that illuminates only a small fraction of thelarge array, a lightweight mechanism for pointing the light pulse to adesired region of the Field-Of-Regard (FOR), a method for reading outonly those pixels illuminated by the light pulse, and a method ofprocessing those data into imagery. Because the laser illuminates only asmall fraction of the large array, the laser power can be small. Becausethe spatial information about the scene is captured by pixels with aknown, precise relative angular position, the laser pointing can be lessprecise compared to the pointing mechanism conventionally used in LIDARsystems. Because the angular width of the light pulse can be 10 times to30 times wider than the receiver angular resolution, the diameter of thelaser scanner can be 10 times to about 30 times smaller than thediameter of the receiver aperture. Because only the pixels illuminatedby the laser are read out, the power consumption of the receiver can beless than what would be otherwise required if the entire array were tobe read out. Because only those pixels containing scene information arepassed to the processing system, the power and resource requirements ofthat processor can be substantially reduced.

The receiver optics in a free-space optical link is typically pointed atthe transmitter. The pointing control accuracy required is normally thegreater of either the diffraction limited angular resolution of thereceiver, λ/D (λ=wavelength, D=aperture diameter), or the field-of-viewof the focal plane array. To allow relaxed pointing control, the focalplane array can be made as large as possible. However, with increasingsize, the power consumed to read out all the pixels can becomesubstantial. The approach described in this section allows only thepixels of interest to be read out, thereby allowing the focal planearray size to be scaled up while maintaining a relatively low powerconsumption.

To facilitate a quantitative description of the approach, the followingterms are defined, without being bound by any particular theory or modeof operation.

An image size describes the angular subtense, as viewed form the LIDARsensor, of the object being imaged. Alternatively, the image size may beunderstood as the number of spatial resolution elements across theobject. For example, suppose an object of a characteristic size S isbeing imaged from an airborne platform at a range R. The angularsubtense is then tan⁻¹ S/R. Given a desire to resolve features of sizef, the image size is N_(f)=S/f. The number of resolution elements in theimage is N_(f) ².

The size of the focal plane array is characterized by the number ofpixels along a linear dimension, N_(x). For a square array the number ofpixels is N_(x) ². The angular subtense of each pixel φ, multiplied bythe number of pixels, gives the total array field-of-view (FOV):Φ=N_(x)φ.

The field-of-regard (FOR) is the total angular width over which thesensor can be pointed. A typical LIDAR can be scanned over an angularwidth θ=±20°.

The term “flood illumination” is typically employed to describe thesituation in which the majority of the pixels are illuminated by thelaser and the laser and focal plane are pointed in concert. If only asmall fraction of the pixels are illuminated by the laser, the term“spot illumination” is used.

LIDAR systems can be categorized in terms of the relative values ofthese size scales, including the typical image size N_(f) this isdesired, the typical angular width θ of the scene, and the total arrayfield of view Φ, as summarized in Table 1. The approach described inthis section can improve cases wherein Φ>N_(f), i.e., the focal planeFOV is wider than the desired image size. Improvement occurs at leastbecause the laser illumination, detector readout, and processingresources can be devoted proportionately to just those parts of the FOVrequiring more signal integration. Also, the receiver does not need tobe slewed to the region of interest since its focal plane field-of-viewis wide enough to cover the entire scene. Existing LIDAR systems ofthese types (i.e. employing a focal plane array of detector pixels)normally employ flood illumination, whereas the approach here employsspot illumination. Because of the power savings allowed by this approachwith a configurable area of interest, it is feasible to scale up thesize of the detector array, thereby eliminating the need for a scanningsystem on the receiver (the signal pulses can still be steered to thedesired region of interest, but the steering can be performed with lowerprecision and with a smaller scanner, at least because the beam issmaller).

TABLE 1 Categorization of LIDAR Systems Pointing/ Allowable TypicalScanning Integration Relative Size Application Requirements Time N_(f) >Φ > θ Wide-area No scanning; high Medium mapping: precision pushbroomknowledge N_(f) > θ > Φ Wide-area High precision Short mapping: scannedknowledge θ > N_(f) > Φ Targeted imaging: High precision Short scannedknowledge θ > Φ > N_(f) Targeted imaging: Low precision Long staringknowledge Φ > θ > N_(f) Cued interrogation Low precision Long knowledgeΦ > N_(f) > θ Cued interrogation None Long

To facilitate the implementation of the LIDAR techniques withconfigurable field of interest, the detector readout integrated circuit(ROIC) can include one or more of the following features: 1) precisiontiming of photon arrivals is only measured in a sub-region of the array,thereby reducing total power consumption; 2) time-gated photon countingis conducted in the non-ROI areas of the detector; the reduced bandwidthrequirements of these circuits allows the circuits to be run at lowervoltage, and therefore reduced leakage current and power; and 3) thetime-gated photon counting can be used to maintain track on the signallaser beam and select the pixel ROI for precision timing.

Optical Field-of-View (FOV) Multiplexer

Optical imaging systems, including LIDAR systems, generally focus lightonto an array of detectors. If the desired system field of view (FOV) iswider than a single detector array FOV, then multiple arrays arecombined in some way.

One way for combining multiple FPA FOVs in an imaging system is based onplacing multiple FPAs in the image plane. One problem with this methodis the gap between the FPAs: either the arrays must be abuttable orthere are losses and image artifacts in the seams. For light-starvedapplications in which all the light collected by the aperture should beexploited (such as active illumination systems or low-light imagers),these losses can be undesirable or even unacceptable. In principle anydetector FPA can be made four-side abuttable, but the associatedengineering challenges are substantial.

Another method to combine multiple FPA FOVs is scanning. An apertureilluminating a single FPA can be scanned over a wide field of regard(FOR) while recording precision pointing measurements. Inpost-processing the measurements are stitched together into a mosaic. Inthis method the integration time per unit solid angle is reduced toaccommodate the scanning. Additionally, the scanning mechanism typicallyneeds to be as precise as the intended image resolution. Finally, inactive illumination systems the laser beam is matched to the single FPAFOV and co-scanned. The beam divergence is tighter and eye safetyconsiderations are more severe.

These challenges can be addressed, at least partially, by an optical FOVmultiplexer which uses an optical system to deflect the image fromvarious portions of the focal plane to detector arrays placed far apartfrom each other. Additionally, various portions of the image can berouted to different optical systems for purposes such as polarimetry,foveated imaging, and dynamic range accommodation.

An optical system focuses light onto a focal plane. One example can be asimple convex lens as shown FIG. 7. The optical system 700 includes afocusing optic 710 (i.e., convex lens in this example) that focuses anincident beam 701 onto a detector array 720 centered on the optical axis706 and located at the focal plane of the lens 705. This optical systemcan be characterized by an f-number, N=f/D, where f is the effectivefocal length of the focusing optic 710 and D is the effective diameterof the focusing optic 710. For an object at infinity, the image isformed at the focal plane 705, a distance f from the lens. The focus isconsidered good near the focal plane within the so-called depth offocus, Δ=±2λN². Typically a focal plane array (FPA) 720 of detectors isplaced at the focal plane; this array can be characterized by a numberof pixels in each of two directions (e.g. N_(x) and N_(y)) and a pixelpitch in each of two directions (e.g. Λ_(x) and Λ_(y)). The angular FOVof an array centered on the optical axis is then limited to Φ=2tan⁻¹(N_(x) Λ_(x)/2f). In order to image a wide FOV, a large FPA (largeN_(x)) can be helpful. Large FPAs are often difficult to manufacture andare expensive, whereas several smaller FPAs are more readily available.It is therefore often desirable to combine together several smaller FPAswithin the same system. If the arrays are abuttable, (i.e., a pixel onone device can be placed immediately next to the pixel of a neighboringdevice without empty space), then smaller arrays can be tiled togetherin the focal plane. However, abuttable detector arrays are often notavailable.

FIG. 8 shows an optical system including an optical FOV multiplexer withan increased FOV by using additional non-abuttable FPAs withoutsuffering seam losses. The system 800 includes a focusing optic 810,such as a lens or curved mirror, which focuses an incident beam 801toward the focal plane of the focusing optic 810. A set of splittingoptic 830 is disposed at the focal plane of the focusing optic 810 tosplit the beam 801 into three components: a first component 801 a, asecond component 801 b, and a third component 801 c. The first component801 a and the second component 801 b are reflected toward a firstdetector array 820 a and a second detector array 802 b, respectively.The third component 801 c is transmitted through the splitting optic 830and propagating toward a third detector array 820 c. Each component (801a-801 c) is transmitted through a respective refocusing optic 840 a-840c so as to be properly focused on the respective detector array.

The splitting optic 830 as shown in FIG. 8 includes a plurality ofdeflecting elements, e.g., prisms or wedge mirrors in this example. Eachprism or wedge mirror can have a size smaller than (N_(x)Λ_(x)) and isplaced at the focal plane 805 of the focusing optic 810. In some cases,the prisms or wedge mirrors may be aligned and/or tilted with respect tothe focal plane in order to reduce misfocus and other aberrations in theimages sensed by the detector arrays 820 a-820 c.

In order to deflect the light out of the incident light cone, thedeflection angle α 806 a and 806 b with respect to the optical axis 807can be greater than the width of the light cone focusing onto the focalplane: α≥α_(min). In the limit that the distance between the opticalaxis and the deflecting element is negligible, α_(min)=2 tan⁻¹(D/2f)=2tan⁻¹(½N). In order to suppress vignetting of adjacent image tiles, thedeflecting element may not protrude more than a distance Δ from thefocal plane at a wedge edge. For the configuration in FIG. 8 with onlytwo wedges, the edges closest to the optical axis can be placed at thefocal plane.

For more general arrangements with many wedges, the non-vignettingcondition may lead to a limitation on the angular width of the lightcone. Consider a flat mirror of size (N_(x) Λ_(x)), deflecting at anangle α_(min). If the center of the mirror is placed on the focal plane,each edge will protrude by approximately (N_(x)Λ_(x))α/2. Using αmin anda small angle approximation, the minimum system f-number can beN_(min)=(N_(x)Λ_(x)/8λ)^(1/3). Inserting numbers for a typical smalldetector FPA with (N_(x)Λ_(x))=10 mm, and operating wavelength λ=1 μm,N_(min) can then be derived as 11. To relax this f-number constraint, asingle flat mirror can be replaced with a stepped mirror (e.g., similarto a Fresnel lens), with each facet at the desired angle but muchshorter. Alternatively, some vignetting may be tolerated. Once the lighthas been deflected out of the incident light cone, it is then refocusedonto the intended detector FPA.

FIG. 9 shows a sensor 900 that includes detector arrays 931 a-931 g(collectively, detector arrays 931) that are tiled across a laserillumination pattern 904 that has been imaged onto a focal plane.Compared to a system with a single detector array, the sensor FOV shownin FIG. 9 can be 3 times larger in each direction, allowing slower scanspeeds. The center array outline is the through-hole in the splittingoptic. Since the center of the laser beam typically has a highintensity, saturation effects in the center detector 931 d array can beaddressed by, for example, inserting a 50/50 beam splitter (not shown)in the collimated space between the refocusing optics to divide thelight onto two detector arrays.

In one example, the refocusing optics 840 a to 840 c can have the samefocal lengths (magnification). In another example, the focal lengths(magnifications) of the refocusing optics 840 a-840 c can be different,thereby allowing foveated imaging. For example, a central portion (e.g.,801 a) of the FOV can be imaged at higher spatial resolution thanoutlying regions (e.g., 801 b and 801 c). This configuration can behelpful for actively illuminated scenes in which the center of theillumination beam is typically brighter than the edges, therebyproducing higher signal rates.

In one example, different portions of the beams 801 a to 801 c arerefocused onto detector arrays 820 a to 820 c. In another example,different portions of the beams 801 a to 801 c can be transmitted tovarious optical trains, such as beam splitters to spread light overmultiple detector FPAs to avoid saturation, multi-spectral analyzers,polarimetric analyzers, and beam splitters to merge with a localoscillator for coherent detection, among others.

Data Processing Flow in LIDAR

FIG. 10 illustrates a data processing flow for 3D imaging using asuitable processing system, such as a LIDAR system like the one shown inFIGS. 1A and 1B. Various modules can be used for various steps in theprocessing. The data processing flow 1000 shown in FIG. 10 includes sixtypes of operations 1010, 1020, 1030, 1040, 1050, and 1060, which aretypically linked together in a linear fashion. The choice of processused to accomplish each step can depend on the desired product quality,the anticipated signal level, the anticipated type of scene, theavailable computational resources, and the level of compression needed.

At step 1010 of the flow 1000, raw timing information is converted intoa 3D coordinate system, which is the natural basis for scene geometryestimation. In one example, the processing system can assume staticalignment during the burst interval, and index the cross-rangecoordinates by pixel row and column, and the line of sight distance(range) by clock counts of the timing logic. In another example, theprocessing system can account for relative motion between the sensor andcorrect for a linear (or higher order) time dependence in both range andcross-range offset when converting the raw timing information to the 3Dcoordinate system. These corrections can be based on an ancillary sensorinputs (inputs which can have lower accuracy requirements), or upon aniterative approach that can maximize image sharpness at later steps(such as 1020, 2040, 1050, 1060, etc.).

Step 1020 of the flow 1000 is to bin the photon detections into a 3Dgrid, forming a 3D histogram. Each photon detection can be attributed toa voxel. Alternatively, one detection may be partially attributed tomultiple voxels so as to reduce 3D quantization artifacts. In oneexample, the processing system can input the row r, column c, and rangebin b from step 1010 and increment the counter for the appropriate voxel(r, c, b). In another example, the processing system can convertfractional values (r, c, b) such as would be generated if a motioncompensation process were used in step 1010. In yet another example, theprocessing system can increment a voxel counter by, for illustrativepurposes only, 8 counts if the input (r, c, b) falls at the center ofthe voxel, but if the (r, c, b) is near the edge of a voxel the 8 countswould be distributed in a weighted fashion between multiple voxels. Inthis example, if the input (r, c, b) falls near the corner of a voxel,the eight nearest neighboring voxels would each be incremented by 1.This method can reduce spatial quantization effects at the expense ofrequiring 3 (i.e. log₂(8)) more bits of resolution per voxel. Inaddition to histogramming photon detections, it can also be advantageousto use 3D histograms to tabulate a) the number of times a detector waslooking through a voxel and could have fired; b) the dark-count orbackground-count rate of the pixels which looked through a voxel; c) therelative illumination intensity for the pixels which looked through avoxel.

After formation of the 3D histogram using photons detected during aburst of pulses, the next step 1030 passes a matched filter through thehistogram. The objective is to optimize subsequent peak detectionprocess 1040 when the system range and/or cross-range impulse responsefunctions are non-Gaussian. In one example, this step 1030 can beoptional (i.e., there can be no filtering). In another example, theprocessing system can use a 1D filter in the range direction. The rangeimpulse response function can be skewed due to the non-symmetric laserpulse and/or the Geiger-mode detection physics. Alternatively,cross-range convolution might be used to correct a non-Gaussian opticalpoint spread function, finite size effects of the detector pixels,cross-talk between detector pixels, etc. A full 3D point spread functionmight be used in the convolution step, incorporating the measured systemresponse. The computation complexity of step 1030 can be weighed againstthe improvements achieved downstream in the peak-finding step 1040. Insome cases the system SWaP cost of the additional computationalcomplexity can be greater than the system SWaP cost of increased signallevels that might be achieved with, for example, a brighter laser, alarger aperture, or a longer integration time.

The peak detection step 1040 of the method 1000 identifies groups ofphoton detections within the 3D histogram that are statisticallysignificant, and are likely not background noise. There can be atradeoff between a high probability of detection and a low false-alarmrate. In general, tabulating more information in step 1020 (such as darkcount probability, number of interrogations, illumination intensity)allows more advanced peak detection processes and better ReceiverOperating Characteristic (ROC) performance (higher probability ofdetection at lower false alarm rates). In one example, a peak detectorcan look for the single largest peak in the range dimension that exceedsa fixed threshold value (number of detected photons). In anotherexample, the processing system can look for multiple peaks in the rangedirection, identifying all that exceed a given threshold. In yet anotherexample, the processing system can adjust the threshold in both therange and the cross-range direction, accommodating the varying number ofinterrogations, the varying pixel illumination, and the varyingbackground count rate. Peak detectors might be optimized to find, forexample, unobscured surfaces, unobscured linear features such assuspended wires, and partially-obscured objects.

Once significant peaks in the 3D histogram are identified, the PeakEstimation 1050 involves estimating the most likely 3D location of theobject that gives rise to the peak. In one example, a group of histogramvoxels in the immediate vicinity of the peak might be used to shift thereported coordinate of the peak away from the center of a voxel. Thistechnique can improve the apparent flatness of planar surfaces andapparent linearity of linear features such as building edges andsuspended wires. In another example, the processing system can simplyreport the indices of the 3D voxel containing the peak.

At optional step 1060, relative reflectivity of the object can beestimated. Reflectivity information can add to the interpretability ofthe imagery and can be useful when registering together the imagery fromindividual bursts of pulses (e.g., image chips). In one example, anestimate can be based on the occupation of the voxel identified ashaving a significant peak. Since the illumination typically may varyacross the array, it can be helpful to normalize this raw photon countby the pixel illumination. When motion compensation is used in step1010, there might be several pixels that interrogate a given scenevoxel, and therefore the illumination factor for each of these pixelscan be accounted. In the presence of partial obscurations, it can behelpful, when normalizing the photon count, to estimate the amount ofillumination that is blocked by the obscurant, which is generallyproportional to the obscuration fraction seen by each of the detectorpixels. Without being bound by any particular theory or mode ofoperation, the number of photons detected in the canopy voxels isproportional to the product of the canopy reflectivity and thefractional obscuration. In order to estimate the fractional obscuration,a canopy reflectivity can be assumed based upon knowledge of the typicalreflectivities of the types of obscurants in the scene.

In another example, an approach for achieving contrast in intensityimages of partially-obscured scenes is to make repeated observations atvarious wavelengths or polarizations without varying the sensor-scenegeometry. These multiple observations can be made through identicalobscurations, so the fraction of light that is blocked can be common tothe entire set of measurements. Intensity contrast images can beconstructed by taking ratios of the return signal strength for thevarious measurements. If N measurements are collected (for example, ifthe scene is probed using light at 532 nm, 1064 nm, and 1540 nm, thenN=3), then N−1 intensity images can be created. The three techniques(peak value, normalized peak value, and FOPEN normalized peak value) in1060 represent just a few of the multiple approaches to estimatingreflectivity. Taken together, the processing steps 1010 through 1060comprise a powerful means to reduce the data storage volume andtransmission bandwidth needed to communicate the salient features of thescene to subsequent processing and operator computers.

Examples of 3D Imaging

FIGS. 11A-11C show example results for coarse registration of images ofrespective 100 m² areas that overlap with neighboring areas by about 70%and that were acquired by an airborne LIDAR system like the one shown inFIG. 1A. In general, FIGS. 11A-11C show that techniques disclosed herecan be promising to allow 3D image chips to be acquired without pointingknowledge, and to subsequently be stitched back together into a usableimage.

The ability to register individual, small image chips into a largemosaic with sufficient image fidelity can be beneficial to the snapshotimaging concept. FIGS. 11A-11C show a 3D LIDAR dataset of a small urbanarea, processed with spatial resolution of 20 cm. Two chips of edgelength L are excised out of the dataset, chosen to have an overlap areafraction f with each other. One of the chips is offset by a 3D vectors.It is then registered to the other chip using a coarse registrationprocess based on spatial Fourier transforms for illustrating purposesonly. In practice, a fine-resolution processes such as Iterative ClosestPoint (ICP), which can be reliable and convergent for small offsets, canbe used. The process can be repeated for many offset vectors s,tabulating the accuracy of the coarse registration. The registrationperformance can be a weak function of the distance |s| up to the chipsize L, as expected for this Fourier transform approach. The maximumresidual uncorrected offset distance r can be determined for the 70thpercentile, 90th percentile, and 95th percentile of the large number ofrandom offsets s that were tested. The entire process is repeated foranother pair of chips at a different location in the urban scene.Altogether 88 sites are studied on a grid that uniformly samples theurban scene, as depicted by the dots in the maps in FIGS. 11B and 11C.The sites include beaches, streets, buildings, and foliage. The resultsdepicted in FIGS. 11A-11C are based on L=10 m (50 resolution elements)and f=0.7. 60% of the sites register to within about a meter whereas 20%do not register accurately. Most of these sites are in the beach areawhere the absence of 3D structure within the small image chips mayexplain the observation. Other sites with poor registration may haveonly a few edges or surfaces. Incorporation of intensity informationwithin the registration process may improve the results at theseanomalous sites.

CONCLUSION

While various inventive embodiments have been described and illustratedherein, those of ordinary skill in the art will readily envision avariety of other means and/or structures for performing the functionand/or obtaining the results and/or one or more of the advantagesdescribed herein, and each of such variations and/or modifications isdeemed to be within the scope of the inventive embodiments describedherein. More generally, those skilled in the art will readily appreciatethat all parameters, dimensions, materials, and configurations describedherein are meant to be exemplary and that the actual parameters,dimensions, materials, and/or configurations will depend upon thespecific application or applications for which the inventive teachingsis/are used. Those skilled in the art will recognize, or be able toascertain using no more than routine experimentation, many equivalentsto the specific inventive embodiments described herein. It is,therefore, to be understood that the foregoing embodiments are presentedby way of example only and that, within the scope of the appended claimsand equivalents thereto; inventive embodiments may be practicedotherwise than as specifically described and claimed. Inventiveembodiments of the present disclosure are directed to each individualfeature, system, article, material, kit, and/or method described herein.In addition, any combination of two or more such features, systems,articles, materials, kits, and/or methods, if such features, systems,articles, materials, kits, and/or methods are not mutually inconsistent,is included within the inventive scope of the present disclosure.

The above-described embodiments can be implemented in any of numerousways. For example, embodiments of designing and making the 3D imagingsystems disclosed herein may be implemented using hardware, software ora combination thereof. When implemented in software, the software codecan be executed on any suitable processor or collection of processors,whether provided in a single computer or distributed among multiplecomputers.

Further, it should be appreciated that a computer may be embodied in anyof a number of forms, such as a rack-mounted computer, a desktopcomputer, a laptop computer, or a tablet computer. Additionally, acomputer may be embedded in a device not generally regarded as acomputer but with suitable processing capabilities, including a PersonalDigital Assistant (PDA), a smart phone or any other suitable portable orfixed electronic device.

Also, a computer may have one or more input and output devices. Thesedevices can be used, among other things, to present a user interface.Examples of output devices that can be used to provide a user interfaceinclude printers or display screens for visual presentation of outputand speakers or other sound generating devices for audible presentationof output. Examples of input devices that can be used for a userinterface include keyboards, and pointing devices, such as mice, touchpads, and digitizing tablets. As another example, a computer may receiveinput information through speech recognition or in other audible format.

Such computers may be interconnected by one or more networks in anysuitable form, including a local area network or a wide area network,such as an enterprise network, and intelligent network (IN) or theInternet. Such networks may be based on any suitable technology and mayoperate according to any suitable protocol and may include wirelessnetworks, wired networks or fiber optic networks.

The various methods or processes (e.g., of designing and making the 3Dimaging systems disclosed above) outlined herein may be coded assoftware that is executable on one or more processors that employ anyone of a variety of operating systems or platforms. Additionally, suchsoftware may be written using any of a number of suitable programminglanguages and/or programming or scripting tools, and also may becompiled as executable machine language code or intermediate code thatis executed on a framework or virtual machine.

In this respect, various inventive concepts may be embodied as acomputer readable storage medium (or multiple computer readable storagemedia) (e.g., a computer memory, one or more floppy discs, compactdiscs, optical discs, magnetic tapes, flash memories, circuitconfigurations in Field Programmable Gate Arrays or other semiconductordevices, or other non-transitory medium or tangible computer storagemedium) encoded with one or more programs that, when executed on one ormore computers or other processors, perform methods that implement thevarious embodiments of the invention discussed above. The computerreadable medium or media can be transportable, such that the program orprograms stored thereon can be loaded onto one or more differentcomputers or other processors to implement various aspects of thepresent invention as discussed above.

The terms “program” or “software” are used herein in a generic sense torefer to any type of computer code or set of computer-executableinstructions that can be employed to program a computer or otherprocessor to implement various aspects of embodiments as discussedabove. Additionally, it should be appreciated that according to oneaspect, one or more computer programs that when executed perform methodsof the present invention need not reside on a single computer orprocessor, but may be distributed in a modular fashion amongst a numberof different computers or processors to implement various aspects of thepresent invention.

Computer-executable instructions may be in many forms, such as programmodules, executed by one or more computers or other devices. Generally,program modules include routines, programs, objects, components, datastructures, etc. that perform particular tasks or implement particularabstract data types. Typically the functionality of the program modulesmay be combined or distributed as desired in various embodiments.

Also, data structures may be stored in computer-readable media in anysuitable form. For simplicity of illustration, data structures may beshown to have fields that are related through location in the datastructure. Such relationships may likewise be achieved by assigningstorage for the fields with locations in a computer-readable medium thatconvey relationship between the fields. However, any suitable mechanismmay be used to establish a relationship between information in fields ofa data structure, including through the use of pointers, tags or othermechanisms that establish relationship between data elements.

Also, various inventive concepts may be embodied as one or more methods,of which an example has been provided. The acts performed as part of themethod may be ordered in any suitable way. Accordingly, embodiments maybe constructed in which acts are performed in an order different thanillustrated, which may include performing some acts simultaneously, eventhough shown as sequential acts in illustrative embodiments.

All definitions, as defined and used herein, should be understood tocontrol over dictionary definitions, definitions in documentsincorporated by reference, and/or ordinary meanings of the definedterms.

The indefinite articles “a” and “an,” as used herein in thespecification and in the claims, unless clearly indicated to thecontrary, should be understood to mean “at least one.”

The phrase “and/or,” as used herein in the specification and in theclaims, should be understood to mean “either or both” of the elements soconjoined, i.e., elements that are conjunctively present in some casesand disjunctively present in other cases. Multiple elements listed with“and/or” should be construed in the same fashion, i.e., “one or more” ofthe elements so conjoined. Other elements may optionally be presentother than the elements specifically identified by the “and/or” clause,whether related or unrelated to those elements specifically identified.Thus, as a non-limiting example, a reference to “A and/or B”, when usedin conjunction with open-ended language such as “comprising” can refer,in one embodiment, to A only (optionally including elements other thanB); in another embodiment, to B only (optionally including elementsother than A); in yet another embodiment, to both A and B (optionallyincluding other elements); etc.

As used herein in the specification and in the claims, “or” should beunderstood to have the same meaning as “and/or” as defined above. Forexample, when separating items in a list, “or” or “and/or” shall beinterpreted as being inclusive, i.e., the inclusion of at least one, butalso including more than one, of a number or list of elements, and,optionally, additional unlisted items. Only terms clearly indicated tothe contrary, such as “only one of” or “exactly one of,” or, when usedin the claims, “consisting of,” will refer to the inclusion of exactlyone element of a number or list of elements. In general, the term “or”as used herein shall only be interpreted as indicating exclusivealternatives (i.e. “one or the other but not both”) when preceded byterms of exclusivity, such as “either,” “one of,” “only one of,” or“exactly one of” “Consisting essentially of,” when used in the claims,shall have its ordinary meaning as used in the field of patent law.

As used herein in the specification and in the claims, the phrase “atleast one,” in reference to a list of one or more elements, should beunderstood to mean at least one element selected from any one or more ofthe elements in the list of elements, but not necessarily including atleast one of each and every element specifically listed within the listof elements and not excluding any combinations of elements in the listof elements. This definition also allows that elements may optionally bepresent other than the elements specifically identified within the listof elements to which the phrase “at least one” refers, whether relatedor unrelated to those elements specifically identified. Thus, as anon-limiting example, “at least one of A and B” (or, equivalently, “atleast one of A or B,” or, equivalently “at least one of A and/or B”) canrefer, in one embodiment, to at least one, optionally including morethan one, A, with no B present (and optionally including elements otherthan B); in another embodiment, to at least one, optionally includingmore than one, B, with no A present (and optionally including elementsother than A); in yet another embodiment, to at least one, optionallyincluding more than one, A, and at least one, optionally including morethan one, B (and optionally including other elements); etc.

In the claims, as well as in the specification above, all transitionalphrases such as “comprising,” “including,” “carrying,” “having,”“containing,” “involving,” “holding,” “composed of,” and the like are tobe understood to be open-ended, i.e., to mean including but not limitedto. Only the transitional phrases “consisting of” and “consistingessentially of” shall be closed or semi-closed transitional phrases,respectively, as set forth in the United States Patent Office Manual ofPatent Examining Procedures, Section 2111.03.

The invention claimed is:
 1. An apparatus for generating a 3D image of ascene, the apparatus comprising: a light source to illuminate a firstportion of the scene and a second portion of the scene with light pulsesat a repetition rate greater than 10 kHz; a detector array to detectphotons reflected or scattered by the first portion of the scene so asto generate a first detected data set representing respective arrivaltimes and arrival angles of the photons reflected or scattered by thefirst portion of the scene and to detect photons reflected or scatteredby the second portion of the scene so as to generate a second detecteddata set representing respective arrival times and arrival angles of thephotons reflected or scattered by the second portion of the scene; anembedded processor, operably coupled to the detector array, to generatea first processed data set by removing at least one redundant data pointand/or at least one noise data point from the first detected data setand to generate a second processed data set by removing at least oneredundant data point and/or at least one noise data point from thesecond detected data set; and a processor, communicatively coupled tothe embedded processor via a data link, to receive the first processeddata set and the second processed data set and to generate the 3D imageof the scene based on the first processed data set and the secondprocessed data set.
 2. The apparatus of claim 1, wherein the lightsource is an incoherent light source that emits pulses with durations of0.1 ns to 15 ns.
 3. The apparatus of claim 1, wherein the embeddedprocessor is configured to remove the at least one redundant data pointand/or at least one noise data point from the first detected data setvia coincidence processing in an angle-angle range coordinate frame. 4.The apparatus of claim 1, wherein the detector array comprises an arrayof photon-counting detectors.
 5. The apparatus of claim 1, wherein theembedded processor comprises at least one of a Field Programmable GateArray (FPGA) or an Application Specific Integrated Circuit (ASIC). 6.The apparatus of claim 1, wherein the embedded processor is integratedwith the detector array on a substrate.
 7. The apparatus of claim 1,wherein the processor is further configured to estimate a property of asurface of the scene based on the first processed data set, the propertyof the surface comprising at least one of a location, a size, areflectivity, a polarization characteristic, or a motion.
 8. Theapparatus of claim 1, further comprising: a read-out integrated circuit(ROIC), operably coupling the embedded processor to the detector array,to read out photocurrent generated by the detector array, time stampphoton arrivals detected by the detector array, and/or read out pixellocations of the photons detected by the detector array.
 9. Theapparatus of claim 1, further comprising: a wireless link between theembedded processor and the processor to convey the processed data setfrom the embedded processor to the processor.
 10. A method of generatinga representation of a scene, the method comprising: A) illuminating afirst portion of the scene with at least one first light pulse; B)acquiring at least one first raw point cloud from photons reflected orscattered from the first portion of the scene using a focal plane array,a first data point in the at least one first raw point cloudrepresenting a first distance between the focal plane array and a firstscene point in the first portion of the scene; C) removing, with a firstprocessor, at least one of a first noise data point or a first redundantdata point from the at least one first raw point cloud so as to generatea first processed point cloud having fewer data points than the at leastone first raw point cloud; D) conveying the first processed point cloudto a second processor; E) illuminating a second portion of the scenewith a second light pulse; F) acquiring a second raw point cloud fromphotons reflected or scattered from the second portion of the sceneusing the focal plan array, a second data point in the second raw pointcloud representing a second distance between the focal plane array and asecond scene point in the second portion of the scene; G) removing, withthe first processor, at least one of a second noise data point or asecond redundant data point from the second raw point cloud so as togenerate a second processed point cloud having fewer data points thanthe second raw point cloud; H) conveying the second processed pointcloud to the second processor; and I) generating, using the secondprocessor, a three-dimensional image of the scene based at least in parton the first processed point cloud and the second processed point cloud.11. The method of claim 10, wherein the first processed point cloudcomprises a single range value.
 12. The method of claim 10, wherein A)comprises illuminating the first portion of the scene with a firstplurality of light pulses, B) comprises acquiring a first plurality ofraw point clouds from the photons reflected or scattered from the firstportion of the scene using the focal plane array, and C) comprisesgenerating the first processed point cloud based on the first pluralityof raw point clouds via coincidence processing in an angle-angle rangecoordinate frame.
 13. The method of claim 10, wherein B) comprisesacquiring the at least one first raw point cloud using a single photondetector.
 14. The method of claim 10, wherein B) comprises acquiring theat least one first raw point cloud using a Geiger-mode avalanchephotodiode.
 15. The method of claim 10, wherein C) comprises generatingthe first processed point cloud with at least one of a FieldProgrammable Gate Array (FPGA) or an Application Specific IntegratedCircuit (ASIC).
 16. The method of claim 10, wherein C) furthercomprises: estimating a property of a surface of the first portion ofthe scene, the property of the surface comprising at least one of alocation, a size, a reflectivity, a polarization characteristic, and amotion.
 17. The method of claim 10, wherein the first noise data pointoriginates from dark current in the focal plane array and C) comprisesremoving the first noise data point from the at least one first rawpoint cloud.
 18. The method of claim 10, wherein the at least one firstraw point cloud has at least about 10 times as many data points as thefirst processed point cloud.